Struct CrawlerBuilder 

pub struct CrawlerBuilder<K: CrawlerKind> { /* private fields */ }

Builds a Crawler around a crawler kind.

Implementations

impl<K: CrawlerKind> CrawlerBuilder<K>

pub fn new(kind: K) -> Self

Creates a builder with engine defaults.

pub fn request_handler<H: RequestHandler<K::Context>>(self, handler: H) -> Self

Sets the request handler.

pub fn failed_request_handler<H: FailedRequestHandler>(self, handler: H) -> Self

Sets the permanent-failure handler.

pub fn max_concurrency(self, count: usize) -> Self

Sets the maximum number of concurrent requests and pins concurrency to that value.

Calling this after Self::autoscale_mode re-pins the crawler to fixed concurrency. Calling Self::autoscale_mode after this method retains count as the dynamic ceiling.

pub fn min_concurrency(self, count: usize) -> Self

Sets the minimum dynamic concurrency.

pub fn desired_concurrency(self, count: usize) -> Self

Sets the initial desired dynamic concurrency.

pub fn autoscale_mode(self, mode: AutoscaleMode) -> Self

Selects dynamic autoscaling.

Calling this after Self::max_concurrency keeps that value as the dynamic ceiling. Calling Self::max_concurrency afterward re-pins concurrency to a fixed value.

pub fn max_tasks_per_minute(self, count: u32) -> Self

Sets the maximum number of task starts permitted per minute.

pub fn same_domain_delay(self, delay: Duration) -> Self

Sets the minimum delay between requests to the same domain.

pub fn autoscaled_pool_options(self, options: AutoscaledPoolOptions) -> Self

Replaces all autoscaled-pool options for advanced use.

The pool’s task_timeout bounds request preparation, execution, and handler work after an attempt starts. Its maybe_run_interval periodically rechecks the queue as a missed-wakeup safety net and must be greater than zero.

pub fn max_request_retries(self, count: u32) -> Self

Sets the maximum number of ordinary request retries.

pub fn max_session_rotations(self, count: u32) -> Self

Sets the maximum number of session rotations.

pub fn request_handler_timeout(self, timeout: Duration) -> Self

Sets the request-handler timeout.

pub fn internal_operation_timeout(self, timeout: Duration) -> Self

Sets the internal storage-operation timeout.

pub fn configuration(self, configuration: Configuration) -> Self

Sets the resolved crawler configuration.

pub fn storage_client(self, storage: Arc<dyn StorageClient>) -> Self

Overrides the configuration’s storage client.

pub fn request_queue(self, queue: Arc<dyn RequestQueue>) -> Self

Overrides the request queue opened from storage.

This hook lets a crate::sitemap::RequestQueueWithSitemap tandem drive the crawler. Callers resuming a pre-populated persistent queue should pair it with crate::config::ConfigurationBuilder::purge_on_start set to false. Storage still supplies the crawler’s key-value store and other storage objects.

pub fn results_capacity(self, capacity: usize) -> Self

Sets the terminal-result broadcast capacity.

pub fn retry_strategy<S: RetryStrategy>(self, strategy: S) -> Self

Installs an attempt-level retry strategy.

pub fn crawl_policy(self, policy: CrawlPolicy) -> Self

Sets the long-lived link admission and crawl-limit policy.

pub async fn build(self) -> Result<Crawler<K>, CrawlerBuildError>

Builds the crawler and opens its configured storage objects.

The default Configuration purges all data managed by the selected storage client before opening the queue and key-value store. Set purge_on_start(false) to retain existing data.