Crate spire 

Spire

Check out other spire projects here.

The flexible crawler and scraper framework for Rust, built on tokio.

Features

• Tag-based routing: requests carry a tag that picks their handler, with handlers written as async functions of typed extractors.
• Declarative extraction: parse pages with Html, or derive Select to turn CSS selectors into structs.
• A real frontier: deduplication, depth limits, and retries with backoff.
• Polite by default: hosts that ask to slow down get wider gaps, and a Retry-After pauses the host for as long as it asks.
• Concurrent by default: requests run concurrently up to a limit, and the crawl ends once nothing is queued or in flight.
• Two backends: plain HTTP through reqwest, and real browsers through WebDriver for pages that need JavaScript.

Installation

[dependencies]
spire = { version = "0.8.0", features = ["reqwest", "macros"] }

See the book for every feature flag.

Quick Start

Crawl a listing, follow its pagination and extract every product:

use spire::backend::ReqwestBackend;
use spire::dataset::InMemDataset;
use spire::extract::html::Select;
use spire::frontier::seed::Seed;
use spire::frontier::{FrontierExt, InMemFrontier};
use spire::handler::{Collect, FollowLinks};
use spire::crawler::Tags;
use spire::{Crawler, Result, Router};

/// The kinds of page the crawl visits, routed by tag.
#[derive(Debug, Clone, Copy, Tags)]
enum Page {
    Listing,
    Product,
}

#[derive(Debug, Select)]
#[select("main.product")]
struct Product {
    #[select("h1")]
    name: String,
    #[select(".price")]
    price: Option<f64>,
}

async fn crawl() -> Result<()> {
    let router = Router::new()
        // Two handlers on one listing page, fetched once: follow the next
        // page and every product.
        .route(Page::Listing, (
            FollowLinks::new(Page::Listing).select("a.next"),
            FollowLinks::new(Page::Product).select("article.product a"),
        ))
        .route(Page::Product, Collect::<Product>::new())
        .require::<Page>();

    let crawler = Crawler::new(ReqwestBackend::default(), router)
        .with_frontier(InMemFrontier::fifo().dedup().max_depth(5).retry(3).tag_backoff())
        .with_dataset(InMemDataset::<Product>::new())
        .with_concurrency(16)
        .with_seed(Seed::get("https://example.com/products")?.with_tag(Page::Listing));

    let stats = crawler.run().await?;
    let products = crawler.dataset::<Product>()?.read_all().await?;
    println!("{} products in {} requests", products.len(), stats.requests);
    Ok(())
}

FollowLinks, Collect and FollowSitemap cover the common jobs. Anything else is an async function of extractors, e.g. async fn price(uri: Uri, Element(product): Element<Product>) -> Result<()>, and mixes freely with them in a tuple of up to 4 handlers. They run in turn on the same page, and stop at the first that does not continue.

Relative links resolve against the page they were found on, after redirects. Tags are not inherited: a followed link without one goes to the first route_path pattern matching its path, such as /products/*, or else the fallback. require fails at startup if a variant has no route, and a handler serving several variants takes page: Page as an argument. A URL is fetched once whatever its tag. Error statuses become failures: 5xx and 429 are retried, honouring Retry-After.

Feature flags

Defaults are html and rustls-tls; choose an HTTP or browser backend explicitly. full enables every feature below except native-tls and testing. Backend-specific options are enabled on their own crates.

Feature	Enables
html (default)	Html, Element, Links, Meta, JsonLd, Select, Collect and FollowLinks
agent	Typed model extraction with Extract<T>; also enables markdown
markdown	The Markdown extractor; also enables html
macros	Select, FromRef, Tags and Revisit derives; also enables html
reqwest	ReqwestBackend and the reqwest re-export
thirtyfour	ThirtyfourBackend, Tab and the thirtyfour re-export
user-agents	Browser profiles and matching navigation headers for identities
robots	Obeying robots.txt with Crawler::with_robots
sitemap	The Sitemap extractor and FollowSitemap handler
serde	Serializable seeds, fingerprints and Bloom snapshots
json	JsonlDataset; also enables serde
csv	CsvDataset; also enables serde
redb	Crash-safe frontier, seen set and datasets through RedbStore; persistent answer caching with agent
rustls-tls (default)	Rustls TLS for enabled backends
native-tls	Platform-native TLS for enabled backends
testing	Offline testing with MockSite, MockPage, MockSession and Visit
full	Every feature except native-tls and testing

See Installation for backend-specific compression, proxy, cookie and driver-management options.

Guide

The Spire Book explains every part of a crawl:

• Getting started: installation, a first crawl, and how a crawl works
• Handling pages: routing, extractors, state, HTML, and storing results
• Discovering pages: following links, best-first crawling, sitemaps, and avoiding traps
• Fetching: HTTP, browsers, middleware, and identities
• Running a crawl: the frontier, errors and retries, limits and politeness, recrawls, crash safety, and logs and statistics
• Extending spire: testing, custom backends, and custom storage

The API reference is on docs.rs.

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License; see the LICENSE file for details.

Acknowledgments

• Built on tokio for async runtime
• Routing pattern inspired by axum

Modules

agentagent

Model providers, extraction, budgets and caches.

backend

Backends and the clients they hand out, plus the enabled HTTP and browser backends.

crawler

Crawl metadata, settings and statistics; the Crawler itself is at the crate root.

dataset

Datasets: where handlers store what they scrape.

extract

Extractors: typed handler arguments taken from the Context.

frontier

The frontier: requests waiting to be crawled.

handler

Handlers: async functions of extractors that handle a request.

http

HTTP requests and responses, their crawl metadata, and the underlying http crate.

identity

Identities: who a request appears to come from, with its proxy, headers and cookies.

redbredb

redb: a crash-safe frontier, seen set and datasets in one file.

testingtesting

Test handlers offline with MockSite, MockPage and Visit.

worker

Workers, which handle fetched requests, their Outcomes, and the middleware.

Structs

Context

A request being crawled, with everything needed to handle it.

Error

An error carrying a classified ErrorKind, a message, and an optional source.

Router

Routes requests to handlers by their Tag, or else their path.

Enums

ErrorKind

Classification of a failure, so callers can branch without matching on messages.

Type Aliases

Crawler

Crawls requests with a Backend, handling them with a Router by default.

Result

A result whose error defaults to Error.