Struct CrawlerEngine 

pub struct CrawlerEngine<S: Spider> { /* private fields */ }

Runs a Spider to completion (or until paused).

The engine dequeues requests by priority and processes them concurrently up to the spider’s global and per-domain limits. Each request goes through robots.txt checks (when enabled), the allowed-domains filter, the dev-mode cache, politeness delays (static or AutoThrottle), the session fetch, blocked-response detection with retries, and finally the spider’s parse callback — whose items are piped through on_scraped_item and whose follow-up requests are enqueued (deduplicated).

Construct with CrawlerEngine::new, run with CrawlerEngine::crawl, and optionally pause a running crawl with CrawlerEngine::request_pause — a paused crawl writes a checkpoint that the next run restores automatically.

Implementations

impl<S: Spider> CrawlerEngine<S>

pub fn new( spider: Arc<S>, session_manager: SessionManager, crawl_dir: Option<&str>, ) -> Result<Self, FetcherError>

Build an engine for spider.

session_manager provides the HTTP sessions (its "default" session is created here if missing — the reason this can fail with FetcherError). crawl_dir overrides where the dev cache and checkpoints live; None uses .scrapling/<spider name>/.

pub fn request_pause(&self)

Ask a running crawl to pause. The crawl loop stops dispatching, waits for in-flight requests to finish, writes a checkpoint (pending requests plus the dedup set), and returns a CrawlResult with paused == true. The next CrawlerEngine::crawl with the same spider/crawl dir resumes from that checkpoint.

pub async fn crawl(self) -> CrawlResult

Run the crawl to completion (or until paused) and return the scraped items and stats.

Restores a checkpoint if one exists (skipping the start requests), pre-fetches robots.txt for the seed origins when the spider obeys robots.txt, then drives the crawl loop until the queue is empty and no tasks are in flight. On normal completion the checkpoint is cleaned up; on pause it is written (see CrawlerEngine::request_pause).