Struct CrawlerConfig 

pub struct CrawlerConfig {

    pub max_concurrent_downloads: usize,
    pub max_pending_requests: usize,
    pub parser_workers: usize,
    pub max_concurrent_pipelines: usize,
    pub channel_capacity: usize,
    pub output_batch_size: usize,
    pub response_backpressure_threshold: usize,
    pub item_backpressure_threshold: usize,
    pub retry_release_permit: bool,
    pub live_stats: bool,
    pub live_stats_interval: Duration,
    pub live_stats_preview_fields: Option<Vec<String>>,
    pub shutdown_grace_period: Duration,
    pub item_limit: Option<usize>,
}

Core runtime configuration for the crawler.

Fields

max_concurrent_downloads: usize

The maximum number of concurrent downloads.

max_pending_requests: usize

The maximum number of outstanding requests tracked by the scheduler.

parser_workers: usize

The number of workers dedicated to parsing responses.

max_concurrent_pipelines: usize

The maximum number of concurrent item processing pipelines.

channel_capacity: usize

The capacity of communication channels between components.

output_batch_size: usize

Number of requests/items processed per parser output batch.

response_backpressure_threshold: usize

Downloader backpressure threshold for the response channel.

item_backpressure_threshold: usize

Parser backpressure threshold for the item channel.

retry_release_permit: bool

When enabled, retries are scheduled outside the downloader permit path.

live_stats: bool

Enables in-place live statistics updates on terminal stdout.

live_stats_interval: Duration

Refresh interval for live statistics output.

live_stats_preview_fields: Option<Vec<String>>

Optional item fields to show in live-stats preview instead of full JSON.

shutdown_grace_period: Duration

Maximum time to wait for a graceful shutdown before forcing task abort.

item_limit: Option<usize>

Maximum number of scraped items to process before stopping the crawl.

Implementations

impl CrawlerConfig

pub fn new() -> Self

Creates a new CrawlerConfig with default settings.

pub fn with_max_concurrent_downloads(self, limit: usize) -> Self

Sets the maximum number of concurrent downloads.

pub fn with_max_pending_requests(self, limit: usize) -> Self

Sets the maximum number of outstanding requests tracked by the scheduler.

pub fn with_parser_workers(self, count: usize) -> Self

Sets the number of parser workers.

pub fn with_max_concurrent_pipelines(self, limit: usize) -> Self

Sets the maximum number of concurrent pipelines.

pub fn with_channel_capacity(self, capacity: usize) -> Self

Sets the channel capacity.

pub fn with_output_batch_size(self, batch_size: usize) -> Self

Sets the parser output batch size.

pub fn with_response_backpressure_threshold(self, threshold: usize) -> Self

Sets the downloader response-channel backpressure threshold.

pub fn with_item_backpressure_threshold(self, threshold: usize) -> Self

Sets the parser item-channel backpressure threshold.

pub fn with_retry_release_permit(self, enabled: bool) -> Self

Controls whether retry delays release the downloader permit immediately.

pub fn with_live_stats(self, enabled: bool) -> Self

Enables or disables in-place live stats updates on stdout.

pub fn with_live_stats_interval(self, interval: Duration) -> Self

Sets the refresh interval used by live stats mode.

pub fn with_live_stats_preview_fields( self, fields: impl IntoIterator<Item = impl Into<String>>, ) -> Self

Sets which item fields should be shown in live stats preview output.

Field names support dot notation for nested JSON objects, for example: title, source_url, or metadata.Japanese.

You can also set aliases with label=path, for example: url=source_url or jp=metadata.Japanese.

pub fn with_shutdown_grace_period(self, grace_period: Duration) -> Self

Sets the maximum grace period for crawler shutdown.

pub fn with_item_limit(self, limit: usize) -> Self

Sets the maximum number of scraped items to process before stopping the crawl.

pub fn validate(&self) -> Result<(), String>

Validates the configuration.

Trait Implementations

impl Clone for CrawlerConfig

fn clone(&self) -> CrawlerConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlerConfig

fn default() -> Self

Returns the “default value” for a type.