Struct Crawler 

pub struct Crawler<B, W> { /* private fields */ }

Crawls requests from a Frontier with a Backend and a Worker.

run pops seeds from the frontier and handles up to with_concurrency of them at once. Each one gets its own client from the backend and is handed to the worker, whose Outcome is reported back to the frontier.

The run ends once the frontier is empty and no request is in flight, when a worker returns Outcome::Stop, the request or time budget is spent, or the shutdown_token is cancelled. Stopping lets requests in flight finish, for at most the shutdown grace period if one is set.

Politeness is per host: with_host_concurrency and with_host_delay limit how hard a single host is crawled, and with the robots feature, with_robots obeys robots.txt.

Examples

use std::time::Duration;

use spire_core::backend::Noop;
use spire_core::frontier::FrontierExt;
use spire_core::frontier::memory::InMemFrontier;
use spire_core::frontier::seed::Seed;
use spire_core::worker::from_fn;
use spire_core::{Context, Crawler, Result};

async fn handle(cx: Context<Noop>) -> Result<()> {
    cx.queue().follow("https://example.com/next").await?;
    Ok(())
}

let crawler = Crawler::new(Noop, from_fn(handle))
    .with_frontier(InMemFrontier::fifo().dedup().allow_hosts(["example.com"]))
    .with_max_requests(100)
    .with_host_concurrency(2)
    .with_request_timeout(Duration::from_secs(30))
    .with_seed(Seed::get("https://example.com/")?);

let stats = crawler.run().await?;
assert_eq!(stats.requests, 2);

Implementations

impl<B, W> Crawler<B, W>

where B: Backend, W: Worker<B::Client>,

pub async fn run(&self) -> Result<Stats>

Crawls until the frontier is empty and no request is in flight, a worker returns Outcome::Stop, the budget is spent, or the run is shut down.

Errors

Returns an error if a seed cannot be pushed, or if the frontier fails mid-run; requests in flight are drained first.

impl<B, W> Crawler<B, W>

pub fn new(backend: B, worker: W) -> Self

Creates a crawler with an InMemFrontier::fifo frontier that retries with the default RetryPolicy, no datasets, a concurrency of 8 and no limits.

pub fn with_frontier(self, frontier: impl Frontier) -> Self

Replaces the frontier.

Seeds added with with_seed are pushed when the run starts, so they go to the new frontier.

pub fn with_dataset<T, D>(self, dataset: D) -> Self

where T: Send + 'static, D: Dataset<T>,

Registers dataset for items of type T, replacing any previous one.

pub fn with_concurrency(self, limit: usize) -> Self

Sets how many requests are handled at once, at least one.

pub const fn with_max_requests(self, max: usize) -> Self

Stops the run after starting max requests, counting every attempt.

pub const fn with_time_limit(self, limit: Duration) -> Self

Stops the run once it has lasted limit.

pub const fn with_request_timeout(self, timeout: Duration) -> Self

Fails a request that takes longer than timeout to handle, fetching included. Timed out requests are retried.

pub const fn with_shutdown_grace(self, grace: Duration) -> Self

Cancels requests still in flight grace after the run starts stopping, instead of waiting for them indefinitely.

Cancelled requests are reported to the frontier as Report::Cancelled.

pub fn with_host_concurrency(self, limit: usize) -> Self

Sets how many requests to the same host are in flight at once, at least one. Unlimited by default.

pub const fn with_host_delay(self, delay: Duration) -> Self

Waits at least delay between starting two requests to the same host.

pub fn with_robots(self, robots: impl Into<Robots>) -> Self

Available on crate feature robots only.

Obeys robots.txt: fetches each origin’s file before crawling it, skips what it disallows, and honours its Crawl-delay.

Takes a user agent, or a Robots to fetch with another backend or change how long rules are kept; see Robots for how failures are handled. Disallowed requests are reported as Report::Skipped.

use spire_core::backend::Noop;
use spire_core::crawler::Robots;
use spire_core::worker::from_fn;
use spire_core::{Context, Crawler};

let worker = from_fn(|_: Context<Noop>| async {});
let crawler = Crawler::new(Noop, worker).with_robots("MyBot");

// robots.txt through another backend, e.g. HTTP for a browser crawl.
let worker = from_fn(|_: Context<Noop>| async {});
let crawler = Crawler::new(Noop, worker).with_robots(Robots::new("MyBot").with_backend(Noop));

pub fn with_seed(self, seed: Seed) -> Self

Adds a seed to push when the run starts.

pub fn with_seeds(self, seeds: impl IntoIterator<Item = Seed>) -> Self

Adds seeds to push when the run starts.

pub fn backend(&self) -> &B

Returns the backend, e.g. to shut it down after a run.

pub fn queue(&self) -> Queue

Returns a queue for adding seeds to the frontier.

pub fn dataset<T: Send + 'static>(&self) -> Result<Data<T>>

Returns the dataset for items of type T.

Errors

Returns a Dataset error if no dataset is registered for T.

pub fn snapshot(&self) -> Option<Vec<Seed>>

Returns a copy of every seed the frontier holds, see Frontier::snapshot. After a stopped run it includes the requests cancelled or never started.

pub fn stats(&self) -> Stats

Returns the stats of the current run while it runs, or of the last one.

pub fn shutdown_token(&self) -> CancellationToken

Returns a token that stops the run when cancelled.

Requests in flight finish; nothing new starts.

Trait Implementations

impl<B, W> Debug for Crawler<B, W>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.