spire-core

Check out other spire projects here.

The crawl engine behind spire: the crawler, its frontier, backends, workers and datasets.

Overview

• A Backend hands out a Session per request: an HTTP client, a pooled browser session, or anything else that turns a request into a response.
• A Worker handles each request and returns an Outcome: continue, retry, back off, stop or fail.
• The Frontier holds the Seeds waiting to be crawled. Facades add deduplication, a depth limit, retries and per-tag backoff on top of any storage.
• The Crawler runs requests concurrently until the frontier is empty and nothing is in flight.

Crawls are bounded and polite: crawl scope (allow_hosts, filter), request and time budgets, per-request deadlines, per-host concurrency and delays, and robots.txt with the robots feature. With the serde feature, seeds are serializable, so a stopped crawl's frontier and seen set can be snapshotted and restored. JsonlDataset and CsvDataset, with the json and csv features, write results to files.

See the book’s Feature flags page for configuration.

Most users depend on spire, which adds a router, handlers and extractors on top of this crate. Depend on spire-core directly to implement a backend, a frontier or a dataset.

Example

use spire_core::backend::Noop;
use spire_core::frontier::seed::Seed;
use spire_core::frontier::{FrontierExt, InMemFrontier};
use spire_core::worker::{Outcome, from_fn};
use spire_core::crawler::Crawler;
use spire_core::{Context, Result};

async fn handle(cx: Context<Noop>) -> Result<Outcome> {
    let queue = cx.queue().clone();
    let response = cx.response().await?;
    if response.status().as_u16() == 429 {
        return Ok(Outcome::backoff(std::time::Duration::from_secs(30)));
    }

    queue.follow("https://example.com/next").await?;
    Ok(Outcome::Continue)
}

async fn example() -> Result<()> {
    let frontier = InMemFrontier::fifo()
        .dedup()
        .max_depth(3)
        .retry(3)
        .tag_backoff();

    let stats = Crawler::new(Noop, from_fn(handle))
        .with_frontier(frontier)
        .with_concurrency(16)
        .with_seed(Seed::get("https://example.com/")?)
        .run()
        .await?;

    println!("crawled {} requests in {:?}", stats.requests, stats.elapsed);
    Ok(())
}