Skip to main content

InMemFrontier

Struct InMemFrontier 

Source
pub struct InMemFrontier { /* private fields */ }
Expand description

An in-memory Frontier that crawls in insertion, reverse or priority order.

Implementations§

Source§

impl InMemFrontier

Source

pub const fn fifo() -> Self

Crawls seeds in the order they were pushed: breadth-first.

Source

pub const fn lifo() -> Self

Crawls the most recently pushed seed first: depth-first.

Source

pub const fn priority() -> Self

Crawls the seed with the highest priority first, and seeds of equal priority in the order they were pushed.

Trait Implementations§

Source§

impl Debug for InMemFrontier

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for InMemFrontier

Source§

fn default() -> Self

Returns the “default value” for a type. Read more
Source§

impl Extend<Seed> for InMemFrontier

Source§

fn extend<I: IntoIterator<Item = Seed>>(&mut self, seeds: I)

Queues seeds as is, e.g. from a snapshot, without any facade refusing them.

Source§

fn extend_one(&mut self, item: T)

🔬This is a nightly-only experimental API. (extend_one)
Extends a collection with exactly one element.
Source§

fn extend_reserve(&mut self, additional: usize)

🔬This is a nightly-only experimental API. (extend_one)
Reserves capacity in a collection for the given number of additional elements. Read more
Source§

impl FromIterator<Seed> for InMemFrontier

Source§

fn from_iter<I: IntoIterator<Item = Seed>>(seeds: I) -> Self

Collects seeds into a FIFO frontier, e.g. to restore a snapshot.

Source§

impl Frontier for InMemFrontier

Source§

fn report(&self, report: Report) -> impl Future<Output = Result<Ack>> + Send

Re-queues seeds the crawl stopped before finishing, so a snapshot taken after the run includes them.

Source§

fn push(&self, seed: Seed) -> impl Future<Output = Result<Admission>> + Send

Offers a seed, which the frontier may refuse. Read more
Source§

fn pop(&self) -> impl Future<Output = Result<Next>> + Send

Takes the next seed that is ready to be crawled. Read more
Source§

fn size_hint(&self) -> Option<usize>

Returns the number of queued seeds, if known.
Source§

fn snapshot(&self) -> Option<Vec<Seed>>

Returns a copy of every seed held, in no particular order, to save the crawl and resume it later; None if the frontier cannot, e.g. because it persists seeds itself. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<F> FrontierExt for F
where F: Frontier,

Source§

fn dedup(self) -> Dedup<Self>

Refuses seeds seen before, remembering them in memory.
Source§

fn dedup_with<S: SeenSet>(self, seen: S) -> Dedup<Self, S>

Refuses seeds seen before, remembering them in seen.
Source§

fn filter<P>(self, predicate: P) -> Filter<Self, P>
where P: Fn(&Seed) -> bool + Send + Sync + 'static,

Refuses seeds predicate rejects.
Source§

fn allow_hosts<I>( self, hosts: I, ) -> Filter<Self, impl Fn(&Seed) -> bool + Send + Sync + 'static>
where I: IntoIterator<Item: AsRef<str>>,

Refuses seeds outside hosts and their subdomains, e.g. to keep a crawl from following links off-site.
Source§

fn max_depth(self, depth: u32) -> MaxDepth<Self>

Refuses seeds deeper than depth.
Source§

fn retry(self, max_retries: u32) -> Retry<Self>

Retries failed seeds up to max_retries times with exponential backoff.
Source§

fn retry_with(self, policy: RetryPolicy) -> Retry<Self>

Retries failed seeds according to policy.
Source§

fn tag_backoff(self) -> TagBackoff<Self>

Holds back seeds of paused tags.
Source§

fn layer<L: FrontierLayer<Self>>(self, layer: L) -> L::Frontier

Wraps the frontier with a custom facade.
Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more