Struct CrawlSource 

pub struct CrawlSource<F: Fetcher> { /* private fields */ }

A recursive HTTP crawler over a Fetcher.

Implementations

impl<F: Fetcher> CrawlSource<F>

pub fn new(fetcher: F) -> Self

Create a crawler with default configuration.

pub fn with_config(fetcher: F, config: CrawlConfig) -> Self

Create with explicit configuration.

pub fn depth(self, max_depth: u16) -> Self

Set the crawl depth.

pub fn max_pages(self, max_pages: usize) -> Self

Set the page cap.

pub fn scope(self, scope: CrawlScope) -> Self

Set the crawl scope.

pub fn min_delay(self, delay: Duration) -> Self

Set the polite inter-request delay.

pub fn concurrency(self, concurrency: usize) -> Self

Set the maximum number of fetches in flight at once (clamped to ≥ 1).

pub fn max_retries(self, max_retries: u32) -> Self

Set the per-page retry budget for retriable errors.

pub fn respect_robots(self, respect: bool) -> Self

Fetch and honor robots.txt per host (opt-in).

pub fn require_path(self, substring: impl Into<CompactString>) -> Self

Require that every followed link’s URL path contains substring (additive; all required substrings must be present). Confines the crawl to a sub-path.

pub fn accept_extension(self, extension: impl AsRef<str>) -> Self

Only index pages whose URL path ends with extension (additive). Other pages are still crawled for links.

pub fn index_path(self, substring: impl Into<CompactString>) -> Self

Only index pages whose URL path contains substring (additive). Other pages are still crawled for links — this narrows indexing, not crawling.

pub fn validators( self, entries: impl IntoIterator<Item = (UrlKey, CompactString)>, ) -> Self

Provide known validators (If-None-Match) for conditional re-crawls.

pub fn known_edges( self, entries: impl IntoIterator<Item = (UrlKey, Vec<Url>)>, ) -> Self

Provide previously indexed outbound links so 304 pages keep feeding the frontier.

pub fn failed_count(&self) -> u32

Child pages that failed after retries during the most recent crawl.

pub fn take_revalidated(&self) -> Vec<Url>

Drain the pages revalidated as unchanged (HTTP 304) during the most recent crawl. They transferred no body and yielded no Page, so this is the only place their revalidation is visible — a freshness tracker consumes it to record “checked, unchanged” against each URL.

pub fn config(&self) -> &CrawlConfig

Read-only view of the configuration.

Trait Implementations

impl<F: Debug + Fetcher> Debug for CrawlSource<F>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<F: Fetcher> Source for CrawlSource<F>

type Pages<'a> = impl Stream<Item = Result<Page<'a>, Error>> + Send + 'a where Self: 'a

The stream of discovered pages.

fn kind(&self) -> &'static str

A stable label for this source kind ("crawl", "fs").

fn discover<'a>(&'a self, root: &'a SourceRef) -> Self::Pages<'a>

Begin discovery from root.