Struct MaxDepth

Source

pub struct MaxDepth<F> { /* private fields */ }

Expand description

A facade that refuses seeds deeper than a limit.

Implementations§

Source §

impl<F> MaxDepth<F>

Source

pub const fn new(inner: F, max: u32) -> Self

Wraps inner, refusing seeds deeper than max.

Trait Implementations§

Source §

impl<F: Debug> Debug for MaxDepth<F>

Source §

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more

Source §

impl<F: Frontier> Frontier for MaxDepth<F>

Source §

async fn push(&self, seed: Seed) -> Result<Admission>

Offers a seed, which the frontier may refuse. Read more

Source §

fn pop(&self) -> impl Future<Output = Result<NextSeed>> + Send

Takes the next seed that is ready to be crawled. Read more

Source §

fn report(&self, report: Report) -> impl Future<Output = Result<Ack>> + Send

Tells the frontier how a popped seed ended. Read more

Source §

fn size_hint(&self) -> Option<usize>

Returns the number of queued seeds, if known.

Source §

fn snapshot(&self) -> Option<Vec<Seed>>

Returns a copy of every seed held, in no particular order, to save the crawl and resume it later; None if the frontier cannot, e.g. because it persists seeds itself. Read more

Auto Trait Implementations§

§

impl<F> UnwindSafe for MaxDepth<F>
where F: UnwindSafe,

Blanket Implementations§

Source §

impl<T> Any for T
where T: 'static + ?Sized,

Source §

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more

Source §

impl<T> Borrow<T> for T
where T: ?Sized,

Source §

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more

Source §

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source §

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more

Source §

impl<T> From<T> for T

Source §

fn from(t: T) -> T

Returns the argument unchanged.

Source §

impl<F> FrontierExt for F
where F: Frontier,

Source §

fn dedup(self) -> Dedup<Self>

Refuses seeds seen before, remembering them in memory.

Source §

fn dedup_with<S: SeenSet>(self, seen: S) -> Dedup<Self, S>

Refuses seeds seen before, remembering them in seen.

Source §

fn filter(self, predicate: P) -> Filter<Self, P>
where P: Fn(&Seed) -> bool + Send + Sync + 'static,

Refuses seeds predicate rejects.

Source §

fn allow_hosts( self, hosts: I, ) -> Filter<Self, impl Fn(&Seed) -> bool + Send + Sync + 'static>
where I: IntoIterator<Item: AsRef<str>>,

Refuses seeds outside hosts and their subdomains, e.g. to keep a crawl from following links off-site.

Source §

fn max_depth(self, depth: u32) -> MaxDepth<Self>

Refuses seeds deeper than depth.

Source §

fn avoid_traps(self) -> AvoidTraps<Self>

Bounds expanding URL prefixes with conservative, configurable heuristics. Opt in alongside robots.txt and deduplication; legitimate large archives or opaque identifiers may need higher budgets. Read more

Source §

fn avoid_traps_with(self, policy: TrapPolicy) -> AvoidTraps<Self>

Bounds expanding URL prefixes using policy, see Self::avoid_traps.

Source §

fn retry(self, max_retries: u32) -> Retry<Self>

Retries failed seeds up to max_retries times with exponential backoff.

Source §

fn retry_with(self, policy: RetryPolicy) -> Retry<Self>

Retries failed seeds according to policy.

Source §

fn tag_backoff(self) -> TagBackoff<Self>

Holds back seeds of paused tags.

Source §

fn revisit<T: Revisit>(self) -> Revisits<Self, T>
where Self: Schedule,

Crawls pages again on a schedule: a seed reported done or failed is crawled again after its tag’s Revisit::interval, so the crawl keeps going, waiting for the next revisit once nothing else is queued. Read more

Source §