Struct CrawlerConfig 

pub struct CrawlerConfig {
    pub max_concurrent_downloads: usize,
    pub parser_workers: usize,
    pub max_concurrent_pipelines: usize,
    pub channel_capacity: usize,
    pub live_stats: bool,
    pub live_stats_interval: Duration,
}

Core configuration for the crawler’s concurrency settings.

This struct holds tunable parameters that control the parallelism and throughput of the crawler.

Fields

max_concurrent_downloads: usize

The maximum number of concurrent downloads.

parser_workers: usize

The number of workers dedicated to parsing responses.

max_concurrent_pipelines: usize

The maximum number of concurrent item processing pipelines.

channel_capacity: usize

The capacity of communication channels between components.

live_stats: bool

Enables in-place live statistics updates on terminal stdout.

live_stats_interval: Duration

Refresh interval for live statistics output.

Implementations

impl CrawlerConfig

pub fn new() -> Self

Creates a new CrawlerConfig with default settings.

pub fn with_max_concurrent_downloads(self, limit: usize) -> Self

Sets the maximum number of concurrent downloads.

pub fn with_parser_workers(self, count: usize) -> Self

Sets the number of parser workers.

pub fn with_max_concurrent_pipelines(self, limit: usize) -> Self

Sets the maximum number of concurrent pipelines.

pub fn with_channel_capacity(self, capacity: usize) -> Self

Sets the channel capacity.

pub fn with_live_stats(self, enabled: bool) -> Self

Enables or disables in-place live stats updates on stdout.

pub fn with_live_stats_interval(self, interval: Duration) -> Self

Sets the refresh interval used by live stats mode.

pub fn validate(&self) -> Result<(), String>

Validates the configuration.

Trait Implementations

impl Clone for CrawlerConfig

fn clone(&self) -> CrawlerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlerConfig

fn default() -> Self

Returns the “default value” for a type.