pub struct InMemFrontier { /* private fields */ }Expand description
An in-memory Frontier that crawls in insertion, reverse or priority
order.
Implementations§
Source§impl InMemFrontier
impl InMemFrontier
Trait Implementations§
Source§impl Debug for InMemFrontier
impl Debug for InMemFrontier
Source§impl Default for InMemFrontier
impl Default for InMemFrontier
Source§impl Extend<Seed> for InMemFrontier
impl Extend<Seed> for InMemFrontier
Source§fn extend<I: IntoIterator<Item = Seed>>(&mut self, seeds: I)
fn extend<I: IntoIterator<Item = Seed>>(&mut self, seeds: I)
Queues seeds as is, e.g. from a snapshot,
without any facade refusing them.
Source§fn extend_one(&mut self, item: T)
fn extend_one(&mut self, item: T)
🔬This is a nightly-only experimental API. (
extend_one)Extends a collection with exactly one element.
Source§fn extend_reserve(&mut self, additional: usize)
fn extend_reserve(&mut self, additional: usize)
🔬This is a nightly-only experimental API. (
extend_one)Reserves capacity in a collection for the given number of additional elements. Read more
Source§impl FromIterator<Seed> for InMemFrontier
impl FromIterator<Seed> for InMemFrontier
Source§fn from_iter<I: IntoIterator<Item = Seed>>(seeds: I) -> Self
fn from_iter<I: IntoIterator<Item = Seed>>(seeds: I) -> Self
Collects seeds into a FIFO frontier, e.g. to restore a snapshot.
Source§impl Frontier for InMemFrontier
impl Frontier for InMemFrontier
Source§fn report(&self, report: Report) -> impl Future<Output = Result<Ack>> + Send
fn report(&self, report: Report) -> impl Future<Output = Result<Ack>> + Send
Re-queues seeds the crawl stopped before finishing, so a
snapshot taken after the run includes them.
Source§fn push(&self, seed: Seed) -> impl Future<Output = Result<Admission>> + Send
fn push(&self, seed: Seed) -> impl Future<Output = Result<Admission>> + Send
Offers a seed, which the frontier may refuse. Read more
Auto Trait Implementations§
impl !Freeze for InMemFrontier
impl RefUnwindSafe for InMemFrontier
impl Send for InMemFrontier
impl Sync for InMemFrontier
impl Unpin for InMemFrontier
impl UnsafeUnpin for InMemFrontier
impl UnwindSafe for InMemFrontier
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<F> FrontierExt for Fwhere
F: Frontier,
impl<F> FrontierExt for Fwhere
F: Frontier,
Source§fn dedup_with<S: SeenSet>(self, seen: S) -> Dedup<Self, S>
fn dedup_with<S: SeenSet>(self, seen: S) -> Dedup<Self, S>
Refuses seeds seen before, remembering them in
seen.Source§fn allow_hosts<I>(
self,
hosts: I,
) -> Filter<Self, impl Fn(&Seed) -> bool + Send + Sync + 'static>
fn allow_hosts<I>( self, hosts: I, ) -> Filter<Self, impl Fn(&Seed) -> bool + Send + Sync + 'static>
Refuses seeds outside
hosts and their subdomains, e.g. to keep a
crawl from following links off-site.Source§fn retry(self, max_retries: u32) -> Retry<Self>
fn retry(self, max_retries: u32) -> Retry<Self>
Retries failed seeds up to
max_retries times with exponential
backoff.Source§fn retry_with(self, policy: RetryPolicy) -> Retry<Self>
fn retry_with(self, policy: RetryPolicy) -> Retry<Self>
Retries failed seeds according to
policy.Source§fn tag_backoff(self) -> TagBackoff<Self>
fn tag_backoff(self) -> TagBackoff<Self>
Holds back seeds of paused tags.
Source§fn layer<L: FrontierLayer<Self>>(self, layer: L) -> L::Frontier
fn layer<L: FrontierLayer<Self>>(self, layer: L) -> L::Frontier
Wraps the frontier with a custom facade.