Struct CrawlConfig 

pub struct CrawlConfig {
    pub enabled: bool,
    pub max_pages: usize,
    pub max_depth: usize,
    pub respect_robots: bool,
    pub use_sitemap: bool,
}

HTTP crawl and discovery options (URL targets).

Fields

enabled: bool

Enable same-origin HTTP crawl (off by default for backward compatibility).

max_pages: usize

Maximum pages to fetch when crawling from a URL seed.

max_depth: usize

Maximum link depth from the seed URL.

respect_robots: bool

Honor robots.txt disallow rules when crawling.

use_sitemap: bool

Discover URLs from sitemap.xml at the site root when crawling.

Trait Implementations

impl Clone for CrawlConfig

fn clone(&self) -> CrawlConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CrawlConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.