Struct Queue 

pub struct Queue { /* private fields */ }

A cloneable handle for adding seeds to a crawl’s Frontier.

Handlers get a queue bound to the request they handle, so Queue::follow can resolve relative links and derive the depth of the requests it queues.

Implementations

impl Queue

pub fn new(frontier: impl Frontier) -> Self

Creates a queue over frontier.

pub fn child_of<B>(&self, request: &Request<B>) -> Self

Returns a queue whose follow treats links as found on request.

pub fn rebase(&self, uri: Uri)

Resolves relative links against uri from now on, e.g. the URI a response was served from after redirects. Clones share the base.

pub async fn push<S>(&self, seed: S) -> Result<Admission>

where S: TryInto<Seed>, S::Error: Into<Error>,

Adds seed as is.

Errors

Returns an error if seed cannot be converted into a Seed or the frontier fails.

pub async fn follow(&self, link: impl Into<Link>) -> Result<Admission>

Adds a link found on the current request, one level deeper.

The followed request has the link’s tag, or else the seed’s. Tags are not inherited: an untagged link goes to the router’s path routes or fallback, so pagination tags its links explicitly.

The tag is not part of the Fingerprint: with a deduplicating frontier, a URL already followed under one tag is a Duplicate under any other.

Relative hrefs resolve against the page; fragments are dropped. Links that are not http(s) URLs, such as mailto: or malformed ones, are skipped with Admission::Unsupported. Outside a handler, links must be absolute and keep their depth.

Errors

Returns an error if the frontier fails.

pub async fn follow_all<I>(&self, links: I) -> Result<usize>

where I: IntoIterator<Item: Into<Link>>,

Follows every link, returning how many were queued.

Errors

Returns an error if the frontier fails.

pub fn size_hint(&self) -> Option<usize>

Returns the number of queued seeds, if known.

Trait Implementations

impl Clone for Queue

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Queue

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.