Trait FrontierExt 

pub trait FrontierExt: Frontier + Sized {
    // Provided methods
    fn dedup(self) -> Dedup<Self> { ... }
    fn dedup_with<S: SeenSet>(self, seen: S) -> Dedup<Self, S> { ... }
    fn filter<P>(self, predicate: P) -> Filter<Self, P>
       where P: Fn(&Seed) -> bool + Send + Sync + 'static { ... }
    fn allow_hosts<I>(
        self,
        hosts: I,
    ) -> Filter<Self, impl Fn(&Seed) -> bool + Send + Sync + 'static>
       where I: IntoIterator<Item: AsRef<str>> { ... }
    fn max_depth(self, depth: u32) -> MaxDepth<Self> { ... }
    fn avoid_traps(self) -> AvoidTraps<Self> { ... }
    fn avoid_traps_with(self, policy: TrapPolicy) -> AvoidTraps<Self> { ... }
    fn retry(self, max_retries: u32) -> Retry<Self> { ... }
    fn retry_with(self, policy: RetryPolicy) -> Retry<Self> { ... }
    fn tag_backoff(self) -> TagBackoff<Self> { ... }
    fn revisit<T: Revisit>(self) -> Revisits<Self, T>
       where Self: Schedule { ... }
    fn score<S: Score>(self, scorer: S) -> Scored<Self, S> { ... }
    fn layer<L: FrontierLayer<Self>>(self, layer: L) -> L::Frontier { ... }
}

Chains facades onto a Frontier.

Provided Methods

fn dedup(self) -> Dedup<Self>

Refuses seeds seen before, remembering them in memory.

fn dedup_with<S: SeenSet>(self, seen: S) -> Dedup<Self, S>

Refuses seeds seen before, remembering them in seen.

fn filter<P>(self, predicate: P) -> Filter<Self, P>

where P: Fn(&Seed) -> bool + Send + Sync + 'static,

Refuses seeds predicate rejects.

fn allow_hosts<I>( self, hosts: I, ) -> Filter<Self, impl Fn(&Seed) -> bool + Send + Sync + 'static>

where I: IntoIterator<Item: AsRef<str>>,

Refuses seeds outside hosts and their subdomains, e.g. to keep a crawl from following links off-site.

fn max_depth(self, depth: u32) -> MaxDepth<Self>

Refuses seeds deeper than depth.

fn avoid_traps(self) -> AvoidTraps<Self>

Bounds expanding URL prefixes with conservative, configurable heuristics. Opt in alongside robots.txt and deduplication; legitimate large archives or opaque identifiers may need higher budgets.

Put this outside retry/revisit facades to check queued seeds too.

fn avoid_traps_with(self, policy: TrapPolicy) -> AvoidTraps<Self>

Bounds expanding URL prefixes using policy, see Self::avoid_traps.

fn retry(self, max_retries: u32) -> Retry<Self>

Retries failed seeds up to max_retries times with exponential backoff.

fn retry_with(self, policy: RetryPolicy) -> Retry<Self>

Retries failed seeds according to policy.

fn tag_backoff(self) -> TagBackoff<Self>

Holds back seeds of paused tags.

fn revisit<T: Revisit>(self) -> Revisits<Self, T>

where Self: Schedule,

Crawls pages again on a schedule: a seed reported done or failed is crawled again after its tag’s Revisit::interval, so the crawl keeps going, waiting for the next revisit once nothing else is queued.

It goes directly on the storage, which keeps the schedule, e.g. InMemFrontier::fifo().revisit::<Page>().dedup().

fn score<S: Score>(self, scorer: S) -> Scored<Self, S>

Adds scorer’s score to the priority of every seed pushed, so a frontier that orders by priority crawls the best seeds first.

The score adds to the seed’s own priority, e.g. one a handler gave a link, and stacked scorers add up. Seeds are scored once, when pushed: retries and revisits keep their priority. A FIFO or LIFO frontier ignores priorities.

use spire_core::frontier::score::{Keywords, ShallowFirst};
use spire_core::frontier::{FrontierExt, InMemFrontier};

let frontier = InMemFrontier::priority()
    .dedup()
    .score(Keywords::new(["laptop", "review"]).with_weight(10))
    .score(ShallowFirst);

fn layer<L: FrontierLayer<Self>>(self, layer: L) -> L::Frontier

Wraps the frontier with a custom facade.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl<F: Frontier> FrontierExt for F