Struct CrawlerBuilder 

pub struct CrawlerBuilder<S, D>
where
    S: Spider,
    D: Downloader,
{ /* private fields */ }

A fluent builder for constructing Crawler instances.

CrawlerBuilder provides a chainable API for configuring all aspects of a web crawler, including concurrency settings, middleware, pipelines, and checkpoint options.

Type Parameters

• S: The Spider implementation type
• D: The Downloader implementation type

Example

let builder = CrawlerBuilder::new(MySpider)
    .max_concurrent_downloads(8)
    .max_parser_workers(4);

Implementations

impl<S> CrawlerBuilder<S, ReqwestClientDownloader>

where S: Spider,

pub fn new(spider: S) -> CrawlerBuilder<S, ReqwestClientDownloader>

Creates a new CrawlerBuilder for a given spider with the default ReqwestClientDownloader.

Example

let crawler = CrawlerBuilder::new(MySpider)
    .build()
    .await?;

impl<S, D> CrawlerBuilder<S, D>

where S: Spider, D: Downloader,

pub fn max_concurrent_downloads(self, limit: usize) -> CrawlerBuilder<S, D>

Sets the maximum number of concurrent downloads.

This controls how many HTTP requests can be in-flight simultaneously. Higher values increase throughput but may overwhelm target servers.

Default

Defaults to the number of CPU cores, with a minimum of 16.

pub fn max_parser_workers(self, limit: usize) -> CrawlerBuilder<S, D>

Sets the number of worker tasks dedicated to parsing responses.

Parser workers process HTTP responses concurrently, calling the spider’s parse method to extract items and discover new URLs.

Default

Defaults to the number of CPU cores, clamped between 4 and 16.

pub fn max_concurrent_pipelines(self, limit: usize) -> CrawlerBuilder<S, D>

Sets the maximum number of concurrent item processing pipelines.

This controls how many items can be processed by pipelines simultaneously.

Default

Defaults to the number of CPU cores, with a maximum of 8.

pub fn channel_capacity(self, capacity: usize) -> CrawlerBuilder<S, D>

Sets the capacity of internal communication channels.

This controls the buffer size for channels between the downloader, parser, and pipeline components. Higher values can improve throughput at the cost of increased memory usage.

Default

Defaults to 1000.

pub fn downloader(self, downloader: D) -> CrawlerBuilder<S, D>

Sets a custom downloader implementation.

Use this method to provide a custom Downloader implementation instead of the default ReqwestClientDownloader.

pub fn add_middleware<M>(self, middleware: M) -> CrawlerBuilder<S, D>

where M: Middleware<<D as Downloader>::Client> + Send + Sync + 'static,

Adds a middleware to the crawler’s middleware stack.

Middlewares intercept and modify requests before they are sent and responses after they are received. They are executed in the order they are added.

Example

let crawler = CrawlerBuilder::new(MySpider)
    .add_middleware(RateLimitMiddleware::default())
    .add_middleware(RetryMiddleware::new())
    .build()
    .await?;

pub fn add_pipeline<P>(self, pipeline: P) -> CrawlerBuilder<S, D>

where P: Pipeline<<S as Spider>::Item> + 'static,

Adds a pipeline to the crawler’s pipeline stack.

Pipelines process scraped items after they are extracted by the spider. They can be used for validation, transformation, deduplication, or storage (e.g., writing to files or databases).

Example

let crawler = CrawlerBuilder::new(MySpider)
    .add_pipeline(ConsolePipeline::new())
    .add_pipeline(JsonPipeline::new("output.json")?)
    .build()
    .await?;

pub fn with_checkpoint_path<P>(self, path: P) -> CrawlerBuilder<S, D>

where P: AsRef<Path>,

Sets the path for saving and loading checkpoints.

When enabled, the crawler periodically saves its state to this file, allowing crawls to be resumed after interruption.

Requires the checkpoint feature to be enabled.

pub fn with_checkpoint_interval( self, interval: Duration, ) -> CrawlerBuilder<S, D>

Sets the interval between automatic checkpoint saves.

When enabled, the crawler saves its state at this interval. Shorter intervals provide more frequent recovery points but may impact performance.

Requires the checkpoint feature to be enabled.

pub async fn build( self, ) -> Result<Crawler<S, <D as Downloader>::Client>, SpiderError>

where D: Downloader + Send + Sync + 'static, <D as Downloader>::Client: Send + Sync + Clone, <S as Spider>::Item: Send + Sync + 'static,

Builds the Crawler instance.

This method finalizes the crawler configuration and initializes all components. It performs validation and sets up default values where necessary.

Errors

Returns a SpiderError::ConfigurationError if:

• max_concurrent_downloads is 0
• parser_workers is 0
• No spider was provided to the builder

Example

let crawler = CrawlerBuilder::new(MySpider)
    .max_concurrent_downloads(10)
    .build()
    .await?;

Trait Implementations

impl<S> Default for CrawlerBuilder<S, ReqwestClientDownloader>

where S: Spider,

fn default() -> CrawlerBuilder<S, ReqwestClientDownloader>

Returns the “default value” for a type.