spire-core

Check out other spire projects here.

The crawl engine behind spire: the crawler, its frontier, backends, workers and datasets.

Overview

• A Backend hands out a Client per request: an HTTP client, a pooled browser session, or anything else that turns a request into a response.
• A Worker handles each request and returns an Outcome: continue, retry, back off, stop or fail.
• The Frontier holds the Seeds waiting to be crawled. Facades add deduplication, a depth limit, retries and per-tag backoff on top of any storage.
• The Crawler runs requests concurrently until the frontier is empty and nothing is in flight.

Crawls are bounded and polite: crawl scope (allow_hosts, filter), request and time budgets, per-request deadlines, per-host concurrency and delays, and robots.txt with the robots feature. With the serde feature, seeds are serializable, so a stopped crawl's frontier and seen set can be snapshotted and restored, and JsonlDataset writes results as JSON Lines.

Feature Flags

• robots: fetch and obey robots.txt with Crawler::with_robots.
• serde: serializable Seed and Fingerprint, and JsonlDataset.

Most users depend on spire, which adds a router, handlers and extractors on top of this crate. Depend on spire-core directly to implement a backend, a frontier or a dataset.

Example

use spire_core::backend::Noop;
use spire_core::frontier::FrontierExt;
use spire_core::frontier::memory::InMemFrontier;
use spire_core::frontier::seed::Seed;
use spire_core::worker::{Outcome, from_fn};
use spire_core::{Context, Crawler, Result};

async fn handle(cx: Context<Noop>) -> Result<Outcome> {
    let queue = cx.queue().clone();
    let response = cx.fetch().await?;
    if response.status().as_u16() == 429 {
        return Ok(Outcome::backoff(std::time::Duration::from_secs(30)));
    }

    queue.follow("https://example.com/next").await?;
    Ok(Outcome::Continue)
}

# async fn example() -> Result<()> {
let frontier = InMemFrontier::fifo()
    .dedup()
    .max_depth(3)
    .retry(3)
    .tag_backoff();

let stats = Crawler::new(Noop, from_fn(handle))
    .with_frontier(frontier)
    .with_concurrency(16)
    .with_seed(Seed::get("https://example.com/")?)
    .run()
    .await?;

println!("crawled {} requests in {:?}", stats.requests, stats.elapsed);
# Ok(())
# }