Struct Crawler 

pub struct Crawler<K: CrawlerKind> { /* private fields */ }

A configured crawler using lifecycle behavior supplied by K.

Implementations

impl<K: CrawlerKind> Crawler<K>

pub fn builder(kind: K) -> CrawlerBuilder<K>

Starts building a crawler around the given kind.

pub async fn run( &self, start: impl IntoStartRequests, ) -> Result<FinalStatistics, CrawlError>

Runs the crawl to completion.

A crawler runs at most once; a second call returns a non-retryable error.

pub fn handle(&self) -> CrawlerHandle

Creates a weak handle to this crawler.

pub async fn add_requests( &self, reqs: impl IntoIterator<Item = Request> + Send, ) -> Result<(), CrawlError>

Adds requests and waits until the complete batch has been accepted.

pub fn results(&self) -> ResultStream

Subscribes to terminal request snapshots.

pub fn events(&self) -> EventStream

Subscribes to control-plane crawler events.

pub fn stats(&self) -> StatisticsHandle

Returns the live statistics handle.

pub fn autoscaler_snapshot(&self) -> AutoscalerSnapshot

Returns a snapshot of the concurrency scaler.

pub fn stop(&self)

Signals a graceful drain.

pub fn abort(&self)

Signals immediate cancellation.