Struct AvoidTraps 

pub struct AvoidTraps<F> { /* private fields */ }

A facade that bounds expanding URL branches, see FrontierExt::avoid_traps.

When a budget is exceeded, the whole origin/path prefix is blocked for this facade’s lifetime. Seeds already queued there are skipped on pop and reported inwards as Report::Skipped. In-flight work can finish. Other prefixes and origins continue. Refusals are logged and counted by refused, separately from crawler request totals.

This does not inspect page content, execute CSS, or replace robots.txt and deduplication. Keep this facade outside retry/revisit facades so it also checks the seeds they hand out. Snapshots contain pending seeds, not the heuristic state; persistent storage keeps blocked seeds until they are popped and skipped.

Implementations

impl<F> AvoidTraps<F>

pub fn new(inner: F, policy: TrapPolicy) -> Self

Wraps inner with the URL budgets in policy.

pub fn refused(&self) -> usize

Returns the number of refused pushes and skipped queued seeds.

Trait Implementations

impl<F: Debug> Debug for AvoidTraps<F>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<F: Frontier> Frontier for AvoidTraps<F>

async fn push(&self, seed: Seed) -> Result<Admission>

Offers a seed, which the frontier may refuse.

async fn pop(&self) -> Result<NextSeed>

Takes the next seed that is ready to be crawled.

fn report(&self, report: Report) -> impl Future<Output = Result<Ack>> + Send

Tells the frontier how a popped seed ended.

fn size_hint(&self) -> Option<usize>

Returns the number of queued seeds, if known.

fn snapshot(&self) -> Option<Vec<Seed>>

Returns a copy of every seed held, in no particular order, to save the crawl and resume it later; None if the frontier cannot, e.g. because it persists seeds itself.