Struct CrawlPolicy 

#[non_exhaustive]
pub struct CrawlPolicy {
    pub strategy: EnqueueStrategy,
    pub max_crawl_depth: Option<u32>,
    pub max_requests_per_crawl: Option<u64>,
    pub on_skipped: Option<Arc<dyn SkippedHandler>>,
}

Long-lived limits and URL admission policy applied during a crawl.

The type is non-exhaustive so later phases can add robots handling without breaking callers.

Examples

use millipede_core::link_extraction::{CrawlPolicy, EnqueueStrategy};

let policy = CrawlPolicy::new()
    .strategy(EnqueueStrategy::SameDomain)
    .max_crawl_depth(4)
    .max_requests_per_crawl(10_000);
assert_eq!(policy.max_crawl_depth, Some(4));

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

strategy: EnqueueStrategy

The default relationship required between parent and candidate URLs.

max_crawl_depth: Option<u32>

The maximum child crawl depth, or None for no depth limit.

max_requests_per_crawl: Option<u64>

The maximum number of requests accepted in one crawl, or None for no limit.

on_skipped: Option<Arc<dyn SkippedHandler>>

An optional callback invoked for every skipped URL.

Implementations

impl CrawlPolicy

pub fn new() -> Self

Creates the default crawl policy.

pub fn strategy(self, strategy: EnqueueStrategy) -> Self

Sets the default URL admission strategy.

pub fn max_crawl_depth(self, max_crawl_depth: u32) -> Self

Sets the maximum crawl depth.

pub fn max_requests_per_crawl(self, max_requests_per_crawl: u64) -> Self

Sets the maximum number of requests accepted during the crawl.

pub fn on_skipped<H: SkippedHandler>(self, handler: H) -> Self

Sets the skipped-URL callback.

Trait Implementations

impl Debug for CrawlPolicy

fn fmt(&self, formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlPolicy

fn default() -> CrawlPolicy

Returns the “default value” for a type.