Struct Scheduler 

pub struct Scheduler { /* private fields */ }

Implementations

impl Scheduler

pub fn new( _initial_state: Option<()>, ) -> (Arc<Scheduler>, AsyncReceiver<Request>)

Creates a new Scheduler without checkpoint support and returns a tuple containing the scheduler and a request receiver.

pub async fn snapshot(&self) -> Result<(), SpiderError>

Takes a snapshot of the current state of the scheduler (stub when checkpoint feature is disabled).

pub async fn enqueue_request(&self, request: Request) -> Result<(), SpiderError>

Enqueues a new request to be processed.

pub async fn shutdown(&self) -> Result<(), SpiderError>

Sends a shutdown signal to the scheduler.

pub async fn send_mark_as_visited( &self, fingerprint: String, ) -> Result<(), SpiderError>

Sends a message to the scheduler to mark a URL as visited.

pub async fn send_mark_as_visited_batch( &self, fingerprints: Vec<String>, ) -> Result<(), SpiderError>

Sends a message to the scheduler to mark multiple URLs as visited in batch.

pub fn has_been_visited(&self, fingerprint: &str) -> bool

Checks if a URL has been visited using Bloom Filter for fast preliminary check

pub fn should_enqueue_request(&self, request: &Request) -> bool

Checks if a request should be enqueued by checking if its fingerprint has already been visited

pub fn len(&self) -> usize

Returns the number of pending requests in the scheduler.

pub fn is_empty(&self) -> bool

Checks if the scheduler has no pending requests.

pub fn is_idle(&self) -> bool

Checks if the scheduler is idle (has no pending requests).