Struct UrlQueue 

pub struct UrlQueue { /* private fields */ }

Priority URL queue with deduplication and scope control.

Binary heap-based priority queue that ensures URLs are crawled in priority order. Automatically deduplicates URLs and enforces scope control (allowed/blocked domains and paths).

Examples

use crawlkit_engine::queue::{UrlQueue, ScopeConfig, Priority};
use url::Url;

let queue = UrlQueue::new(ScopeConfig::default());
let url = Url::parse("https://example.com").unwrap();
queue.push(url, 0, Priority::HIGH);
assert_eq!(queue.len(), 1);

Implementations

impl UrlQueue

pub fn new(scope: ScopeConfig) -> Self

Creates a new empty queue with the given scope configuration.

pub fn from_crawl_config(config: &CrawlConfig) -> Self

Creates a queue from a CrawlConfig.

pub fn push(&self, url: Url, depth: usize, priority: Priority) -> bool

Pushes a URL into the queue with the given depth and priority.

Returns true if the URL was added, false if it was a duplicate or rejected by scope control.

pub fn push_with_referrer( &self, url: Url, depth: usize, priority: Priority, referrer: Option<Url>, ) -> bool

Pushes a URL with an optional referrer.

Performs deduplication, depth check, and scope filtering.

pub fn pop(&self) -> Option<QueueEntry>

Pops the highest-priority entry from the queue.

pub fn len(&self) -> usize

Returns the number of entries in the queue.

pub fn is_empty(&self) -> bool

Returns true if the queue is empty.

pub fn seen_count(&self) -> usize

Returns the number of unique URLs seen (including popped ones).

pub fn domain_count(&self, domain: &str) -> usize

Returns the number of URLs discovered for a given domain.

pub fn domains(&self) -> Vec<String>

Returns all unique domains that have been queued.

pub fn peek(&self) -> Option<QueueEntry>

Peeks at the highest-priority entry without removing it.

pub fn drain(&self) -> Vec<QueueEntry>

Drains all entries from the queue.