Struct CrawlConfig 

pub struct CrawlConfig {

    pub max_depth: u16,
    pub scope: CrawlScope,
    pub max_pages: usize,
    pub min_delay: Duration,
    pub max_bytes: Option<u64>,
    pub user_agent: Option<CompactString>,
    pub require_path_contains: Vec<CompactString>,
    pub accept_extensions: Vec<CompactString>,
    pub index_path_contains: Vec<CompactString>,
    pub concurrency: usize,
    pub max_retries: u32,
    pub validators: HashMap<UrlKey, CompactString>,
    pub known_edges: HashMap<UrlKey, Vec<Url>>,
    pub respect_robots: bool,
}

Crawl configuration. Every field is honored; see CrawlSource builder methods.

Fields

max_depth: u16

Maximum link depth from the root (root is depth 0).

scope: CrawlScope

Which links to follow.

max_pages: usize

Hard cap on total pages fetched (runaway guard).

min_delay: Duration

Polite delay between fetches.

max_bytes: Option<u64>

Per-page byte ceiling.

user_agent: Option<CompactString>

User-Agent override.

require_path_contains: Vec<CompactString>

A followed link’s URL path must contain all of these substrings (empty = no constraint). Composes with CrawlConfig::scope: use it to confine a crawl to a sub-path that the host’s URL scheme splits across segments (e.g. GitHub’s /tree/ directories and /blob/ files both under /main/docs).

accept_extensions: Vec<CompactString>

Only index pages whose URL path ends with one of these extensions, e.g. ["md"] (empty = index every indexable kind). Non-matching pages are still crawled for links — only their indexing is filtered.

index_path_contains: Vec<CompactString>

Only index pages whose URL path contains all of these substrings (empty = no constraint). Independent of CrawlConfig::require_path_contains: follow broadly, index narrowly — e.g. follow GitHub /tree/ dirs but index only the canonical /blob/ file views, collapsing the /raw/,/commits/ duplicates.

concurrency: usize

Maximum fetches in flight at once (clamped to at least 1). Higher values crawl faster; min_delay still paces request starts globally.

max_retries: u32

Retry budget per page for retriable errors (timeouts, 429/5xx). Honors Retry-After on rate limits; other retries back off exponentially.

validators: HashMap<UrlKey, CompactString>

Known entity tags by canonical-URL key: fetches send If-None-Match and a 304 skips the body transfer entirely (conditional re-crawl).

known_edges: HashMap<UrlKey, Vec<Url>>

Known outbound links by canonical-URL key: when a page revalidates as unchanged (no body to parse), its previously indexed children keep the frontier going, so coverage never shrinks on a 304.

respect_robots: bool

Fetch and honor robots.txt per host (opt-in; requires the robots feature).

Trait Implementations

impl Clone for CrawlConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlConfig

fn default() -> Self

Returns the “default value” for a type.