Struct AdaptiveStrategy 

pub struct AdaptiveStrategy { /* private fields */ }

Adaptive crawling strategy that stops when content saturation is detected.

Tracks the number of new unique terms discovered per page. When the rate of new term discovery drops below a configurable threshold (term saturation), the strategy signals the engine to stop crawling.

Implementations

impl AdaptiveStrategy

pub fn new(window_size: usize, saturation_threshold: f64) -> Self

Create a new adaptive strategy.

• window_size: Number of recent pages for saturation window (default: 10)
• saturation_threshold: Stop when new terms per page ratio drops below this (default: 0.05)

pub fn record_page(&self, html: &str)

Record a page’s content for saturation tracking.

Trait Implementations

impl CrawlStrategy for AdaptiveStrategy

fn select_next(&self, candidates: &[FrontierEntry]) -> Option<usize>

Select the next URL to crawl from a set of candidates. Returns the index into candidates, or None if none should be selected.

fn score_url(&self, _url: &str, depth: usize) -> f64

Score a URL for prioritisation.

fn should_continue(&self, stats: &CrawlStats) -> bool

Whether the crawl should continue given current stats.

fn on_page_processed(&self, page: &CrawlPageResult)

Called after each page is processed. Used by adaptive strategies to track content.

impl Debug for AdaptiveStrategy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for AdaptiveStrategy

fn default() -> Self

Returns the “default value” for a type.