Crate spire_core 

spire-core

Check out other spire projects here.

The crawl engine behind spire: the crawler, its frontier, backends, workers and datasets.

Overview

• A Backend hands out a Session per request: an HTTP client, a pooled browser session, or anything else that turns a request into a response.
• A Worker handles each request and returns an Outcome: continue, retry, back off, stop or fail.
• The Frontier holds the Seeds waiting to be crawled. Facades add deduplication, a depth limit, retries and per-tag backoff on top of any storage.
• The Crawler runs requests concurrently until the frontier is empty and nothing is in flight.

Crawls are bounded and polite: crawl scope (allow_hosts, filter), request and time budgets, per-request deadlines, per-host concurrency and delays, and robots.txt with the robots feature. With the serde feature, seeds are serializable, so a stopped crawl’s frontier and seen set can be snapshotted and restored. JsonlDataset and CsvDataset, with the json and csv features, write results to files.

Feature flags

No features are enabled by default.

Feature	Enables
robots	Fetching and obeying robots.txt with Crawler::with_robots
serde	Serialization of Seed, Fingerprint and BloomSnapshot
json	JsonlDataset, writing JSON Lines; also enables serde
csv	CsvDataset, writing CSV; also enables serde
testing	MockSite, MockPage, MockSession and Visit for offline tests
user-agents	Identity::with_profile and the uagent Profile, UserAgents, Browser and Platform re-exports

Most users depend on spire, which adds a router, handlers and extractors on top of this crate. Depend on spire-core directly to implement a backend, a frontier or a dataset.

Example

use spire_core::backend::Noop;
use spire_core::frontier::seed::Seed;
use spire_core::frontier::{FrontierExt, InMemFrontier};
use spire_core::worker::{Outcome, from_fn};
use spire_core::crawler::Crawler;
use spire_core::{Context, Result};

async fn handle(cx: Context<Noop>) -> Result<Outcome> {
    let queue = cx.queue().clone();
    let response = cx.response().await?;
    if response.status().as_u16() == 429 {
        return Ok(Outcome::backoff(std::time::Duration::from_secs(30)));
    }

    queue.follow("https://example.com/next").await?;
    Ok(Outcome::Continue)
}

async fn example() -> Result<()> {
    let frontier = InMemFrontier::fifo()
        .dedup()
        .max_depth(3)
        .retry(3)
        .tag_backoff();

    let stats = Crawler::new(Noop, from_fn(handle))
        .with_frontier(frontier)
        .with_concurrency(16)
        .with_seed(Seed::get("https://example.com/")?)
        .run()
        .await?;

    println!("crawled {} requests in {:?}", stats.requests, stats.elapsed);
    Ok(())
}

Modules

backend

Backends and the sessions they hand out.

crawler

The Crawler, which drives a crawl to completion.

dataset

Datasets: where handlers store what they scrape.

frontier

The frontier: requests waiting to be crawled.

http

HTTP requests and responses, their crawl metadata, and the underlying http crate.

identity

Identities: who a request appears to come from.

testingtesting

Test handlers offline with MockSite, MockPage and Visit.

worker

Workers, which handle fetched requests, their Outcomes, and the middleware.

Structs

Context

A request being crawled, with everything needed to handle it.

Error

An error carrying a classified ErrorKind, a message, and an optional source.

Enums

ErrorKind

Classification of a failure, so callers can branch without matching on messages.

Type Aliases

BoxError

A boxed, thread-safe source error.

Result

A result whose error defaults to Error.