Struct CrawlerState 

pub struct CrawlerState {
    pub in_flight_requests: Atomic<usize>,
    pub parsing_responses: Atomic<usize>,
    pub processing_items: Atomic<usize>,
}

Represents the shared state of the crawler’s various actors.

This struct provides a centralized mechanism for monitoring the real-time activity of the web crawler. It utilizes atomic counters to keep track of:

• The number of HTTP requests currently in flight (being downloaded).
• The number of responses actively being parsed by spiders.
• The number of scraped items currently being processed by pipelines.

This state information is crucial for determining when the crawler is idle and can be gracefully shut down, or when to trigger checkpointing.

Fields

in_flight_requests: Atomic<usize>

The number of requests currently being downloaded.

parsing_responses: Atomic<usize>

The number of responses currently being parsed.

processing_items: Atomic<usize>

The number of items currently being processed by pipelines.

Implementations

impl CrawlerState

pub fn new() -> Arc<CrawlerState>

Creates a new, atomically reference-counted CrawlerState.

pub fn is_idle(&self) -> bool

Checks if all crawler activities are idle.

Trait Implementations

impl Debug for CrawlerState

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for CrawlerState

fn default() -> CrawlerState

Returns the “default value” for a type.