Struct CrawlConfig 

pub struct CrawlConfig {
    pub max_depth: usize,
    pub preserve_domain: bool,
}

Configuration for recursive crawling.

The crawler first discovers links level-by-level using get_urls, then performs one final batch_fetch over the discovered URL set.

Fields

max_depth: usize

Maximum number of link-expansion steps from the seed URL.

0 means only the seed URL is fetched in the final batch.

preserve_domain: bool

If true, only URLs on the same origin as the seed URL are followed.

In practice this preserves the seed host and effective port, which keeps a crawl pinned to the same site by default.

Trait Implementations

impl Clone for CrawlConfig

fn clone(&self) -> CrawlConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlConfig

fn default() -> Self

Returns the default crawl configuration.

Defaults are intentionally conservative:

• max_depth = 0
• preserve_domain = true