Struct Crawler 

pub struct Crawler<B, W> { /* private fields */ }

Crawls requests from a Frontier with a Backend and a Worker.

run pops seeds from the frontier and handles up to with_concurrency of them at once. Each one gets its own client from the backend and is handed to the worker, whose Outcome is reported back to the frontier.

The run ends once the frontier is empty and no request is in flight, when a worker returns Outcome::Stop, the request or time budget is spent, or the shutdown_token is cancelled. Stopping lets requests in flight finish, for at most the shutdown grace period if one is set.

Politeness is per host: with_host_concurrency and with_host_delay limit how hard a single host is crawled, and with the robots feature, with_robots obeys robots.txt.

Examples

use std::time::Duration;

use spire_core::backend::Noop;
use spire_core::frontier::seed::Seed;
use spire_core::frontier::{FrontierExt, InMemFrontier};
use spire_core::worker::from_fn;
use spire_core::crawler::Crawler;
use spire_core::{Context, Result};

async fn handle(cx: Context<Noop>) -> Result<()> {
    cx.queue().follow("https://example.com/next").await?;
    Ok(())
}

let crawler = Crawler::new(Noop, from_fn(handle))
    .with_frontier(InMemFrontier::fifo().dedup().allow_hosts(["example.com"]))
    .with_max_requests(100)
    .with_host_concurrency(2)
    .with_request_timeout(Duration::from_secs(30))
    .with_seed(Seed::get("https://example.com/")?);

let stats = crawler.run().await?;
assert_eq!(stats.requests, 2);

Implementations

impl<B, W> Crawler<B, W>

where B: Backend, W: Worker<B::Session>,

pub async fn run(&self) -> Result<Stats>

Crawls until the frontier is empty and no request is in flight, a worker returns Outcome::Stop, the budget is spent, or the run is shut down.

Errors

Returns an error if a seed cannot be pushed, or if the frontier fails mid-run; requests in flight are drained first.

impl<B, W> Crawler<B, W>

pub fn new(backend: B, worker: W) -> Self

where B: Backend, W: Worker<B::Session>,

Creates a crawler with an InMemFrontier::fifo frontier that retries with the default RetryPolicy, no datasets, a concurrency of 8 and no limits.

worker must handle the clients of backend, which also lets the compiler infer a router’s client type from the backend.

pub fn with_extension<T: Clone + Send + Sync + 'static>(self, value: T) -> Self

Adds a shared service to every request handled by this crawler.

A later registration of the same type replaces the previous value. Values are cloned for each request; use Arc<T> for shared ownership. Middleware can replace a value on an individual request.

pub fn with_frontier(self, frontier: impl Frontier) -> Self

Replaces the frontier.

Seeds added with with_seed are pushed when the run starts, so they go to the new frontier.

pub fn with_dataset<T, D>(self, dataset: D) -> Self

where T: Send + 'static, D: Dataset<T>,

Registers dataset for items of type T, replacing any previous one.

pub fn with_concurrency(self, limit: usize) -> Self

Sets how many requests are handled at once, at least one.

pub const fn with_max_requests(self, max: usize) -> Self

Stops the run after starting max requests, counting every attempt.

pub const fn with_time_limit(self, limit: Duration) -> Self

Stops the run once it has lasted limit.

pub const fn with_request_timeout(self, timeout: Duration) -> Self

Fails a request that takes longer than timeout to handle, fetching included. Timed out requests are retried.

pub const fn with_shutdown_grace(self, grace: Duration) -> Self

Cancels requests still in flight grace after the run starts stopping, instead of waiting for them indefinitely.

Cancelled requests are reported to the frontier as Report::Cancelled.

pub fn with_host_concurrency(self, limit: impl Into<HostConcurrency>) -> Self

Sets how many requests to the same host are in flight at once: a fixed number, at least one, or a limit per host adapted to how it responds with HostConcurrency::adaptive. Unlimited by default.

pub fn with_host_delay(self, delay: impl Into<HostDelay>) -> Self

Sets the gap between starting two requests to the same host: at least a minimum, a Duration, growing while the host is throttled up to a maximum, one minute unless a range sets it; see HostDelay. No minimum by default.

pub fn with_robots(self, robots: impl Into<Robots>) -> Self

Available on crate feature robots only.

Obeys robots.txt: fetches each origin’s file before crawling it, skips what it disallows, and honours its Crawl-delay.

Takes a user agent, or a Robots to fetch with another backend or change how long rules are kept; see Robots for how failures are handled. Disallowed requests are reported as Report::Skipped.

use spire_core::backend::Noop;
use spire_core::crawler::Robots;
use spire_core::worker::from_fn;
use spire_core::crawler::Crawler;
use spire_core::Context;

let worker = from_fn(|_: Context<Noop>| async {});
let crawler = Crawler::new(Noop, worker).with_robots("MyBot");

// robots.txt through another backend, e.g. HTTP for a browser crawl.
let worker = from_fn(|_: Context<Noop>| async {});
let crawler = Crawler::new(Noop, worker).with_robots(Robots::new("MyBot").with_backend(Noop));

pub fn with_seed(self, seed: Seed) -> Self

Adds a seed to push when the run starts.

pub fn with_seeds(self, seeds: impl IntoIterator<Item = Seed>) -> Self

Adds seeds to push when the run starts.

pub fn with_identities(self, identities: Identities) -> Self

Sends every request as one of identities: through its proxy, with its headers and its own cookies, so each looks like one visitor. robots.txt is fetched as an identity too, unless robots.txt has a backend of its own.

The backend connects each request as its identity, see Backend::connect_as; one that cannot route through a proxy fails the requests of identities that have one. Identities that keep failing are taken out of use, see Identities.

use spire_core::backend::Noop;
use spire_core::identity::{Identities, Identity};
use spire_core::worker::from_fn;
use spire_core::crawler::Crawler;
use spire_core::Context;

let identities = Identities::new([Identity::new("a"), Identity::new("b")]);
let worker = from_fn(|_: Context<Noop>| async {});
let crawler = Crawler::new(Noop, worker).with_identities(identities);

pub fn backend(&self) -> &B

Returns the backend, e.g. to shut it down after a run.

pub fn queue(&self) -> Queue

Returns a queue for adding seeds to the frontier.

pub fn dataset<T: Send + 'static>(&self) -> Result<Data<T>>

Returns the dataset for items of type T.

Errors

Returns a Dataset error if no dataset is registered for T.

pub fn snapshot(&self) -> Option<Vec<Seed>>

Returns a copy of every seed the frontier holds, see Frontier::snapshot. After a stopped run it includes the requests cancelled or never started.

pub fn stats(&self) -> Stats

Returns the stats of the current run while it runs, or of the last one.

pub fn events(&self) -> Events

Subscribes to the Events of every run from now on: requests starting, fetched, handled, retried or skipped, hosts slowed down, robots.txt fetched, and the run stopping and finishing.

Events are not built while no one is subscribed. A subscriber never slows the crawl down: one that falls more than 1024 events behind misses the oldest, and gets an Event::Lagged in their place. The stream ends once the crawler is dropped; Event::Finished marks the end of a run.

use futures::StreamExt;
use spire_core::backend::Noop;
use spire_core::crawler::Crawler;
use spire_core::crawler::event::Event;
use spire_core::frontier::seed::Seed;
use spire_core::worker::from_fn;
use spire_core::{Context, Result};

let worker = from_fn(|_: Context<Noop>| async {});
let crawler = Crawler::new(Noop, worker).with_seed(Seed::get("https://example.com/")?);

let mut events = crawler.events();
let watch = tokio::spawn(async move {
    let mut handled = 0;
    while let Some(event) = events.next().await {
        match event {
            Event::Handled { .. } => handled += 1,
            Event::Finished { .. } => break,
            _ => {}
        }
    }
    handled
});

crawler.run().await?;
assert_eq!(watch.await.unwrap(), 1);

pub fn shutdown_token(&self) -> CancellationToken

Returns a token that stops the run when cancelled.

Requests in flight finish; nothing new starts.

Trait Implementations

impl<B, W> Debug for Crawler<B, W>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.