Struct Frontier 

pub struct Frontier { /* private fields */ }

Holds the URLs still to fetch. Never admits more than max_pages URLs to fetch in total (fetched plus queued), so memory stays flat on endless URL spaces. Redirect targets are remembered past the cap but never queued, so records stay at most 2 × max_pages.

Levels are a barrier: pop only returns URLs of the current level, and the next level becomes current when the caller asks with advance. Sitemap-only URLs come last.

Implementations

impl Frontier

pub fn new(max_pages: u32) -> Frontier

pub fn seed(&mut self, url: Url) -> bool

Queues the start URL at depth 0. False when it was seen already or the cap is reached.

pub fn push_link(&mut self, url: Url, from_depth: u16) -> bool

Queues a link found on a page at from_depth for the next level.

pub fn admit_redirect_target(&mut self, url: &Url) -> bool

Marks a redirect target as seen without queueing or counting it: its record is built from the response that already arrived, so it costs no fetch. The cap does not apply (it bounds fetches), but a URL already seen is refused.

pub fn add_sitemap_urls(&mut self, urls: impl IntoIterator<Item = Url>)

Queues sitemap URLs that nothing linked to; they are fetched after link exploration.

pub fn pop(&mut self) -> Option<Queued>

The next URL of the current level.

pub fn advance(&mut self) -> bool

Starts the next level. When link exploration is over, starts the sitemap-only URLs (once). False when nothing is left.

pub fn requeue_front(&mut self, q: Queued)

Puts a URL back at the front of the current level (a 429/503 retry). It was counted when it was first queued and is not counted again.

pub fn is_seen(&self, url: &Url) -> bool

pub fn capped(&self) -> bool

True once a URL was refused because the cap was reached.

pub fn queued(&self) -> usize

URLs waiting to be fetched, in every queue.

Trait Implementations

impl Debug for Frontier

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.