Struct CrawlStats 

pub struct CrawlStats {

    pub requests_count: u64,
    pub concurrent_requests: u32,
    pub concurrent_requests_per_domain: u32,
    pub failed_requests_count: u64,
    pub offsite_requests_count: u64,
    pub robots_disallowed_count: u64,
    pub cache_hits: u64,
    pub cache_misses: u64,
    pub response_bytes: u64,
    pub items_scraped: u64,
    pub items_dropped: u64,
    pub start_time: f64,
    pub end_time: f64,
    pub download_delay: f64,
    pub blocked_requests_count: u64,
    pub custom_stats: HashMap<String, Value>,
    pub response_status_count: HashMap<String, u64>,
    pub domains_response_bytes: HashMap<String, u64>,
    pub sessions_requests_count: HashMap<String, u64>,
    pub proxies: Vec<String>,
    pub log_levels_counter: HashMap<String, u64>,
}

Aggregate statistics collected during a crawl run.

The crawler engine populates this struct as it processes requests. After the crawl finishes, you can inspect it via CrawlerEngine::stats or from the returned CrawlResult. All counters start at zero and are incremented atomically during the crawl loop.

Fields

requests_count: u64

Total number of requests dispatched.

concurrent_requests: u32

Maximum number of concurrent requests allowed.

concurrent_requests_per_domain: u32

Maximum number of concurrent requests per domain.

failed_requests_count: u64

Number of requests that failed with an error.

offsite_requests_count: u64

Number of requests rejected because their domain was not allowed.

robots_disallowed_count: u64

Number of requests blocked by robots.txt rules.

cache_hits: u64

Number of responses served from the cache.

cache_misses: u64

Number of responses that were not found in the cache.

response_bytes: u64

Total bytes received across all responses.

items_scraped: u64

Number of items successfully scraped.

items_dropped: u64

Number of items dropped by the item pipeline.

start_time: f64

Unix timestamp when the crawl started.

end_time: f64

Unix timestamp when the crawl ended.

download_delay: f64

Configured delay in seconds between consecutive requests.

blocked_requests_count: u64

Number of requests that received a blocked status code.

custom_stats: HashMap<String, Value>

User-defined custom statistics.

response_status_count: HashMap<String, u64>

Count of responses grouped by HTTP status code.

domains_response_bytes: HashMap<String, u64>

Total bytes received grouped by domain.

sessions_requests_count: HashMap<String, u64>

Number of requests dispatched per session.

proxies: Vec<String>

List of proxy addresses used during the crawl.

log_levels_counter: HashMap<String, u64>

Count of log messages grouped by level.

Implementations

impl CrawlStats

pub fn elapsed_seconds(&self) -> f64

Returns the wall-clock duration of the crawl in seconds, computed as end_time - start_time. Both timestamps are Unix epoch seconds recorded at the start and end of CrawlerEngine::crawl.

pub fn requests_per_second(&self) -> f64

Returns the average number of requests completed per second over the entire crawl. Returns 0.0 if the crawl duration was zero (e.g., an instant crawl with no network calls).

pub fn increment_status(&mut self, status: u16)

Increments the counter for the given HTTP status code. Status codes are stored under keys like "status_200" or "status_404" in the response_status_count map, making it easy to spot error patterns.

pub fn increment_response_bytes(&mut self, domain: &str, count: u64)

Adds count bytes to both the global response_bytes total and the per-domain counter in domains_response_bytes. This is called by the engine after every successful fetch so you can identify bandwidth-heavy domains.

pub fn increment_requests_count(&mut self, sid: &str)

Increments the total requests_count and the per-session counter in sessions_requests_count. The engine calls this before every fetch so you can see how load is distributed across sessions.

Trait Implementations

impl Clone for CrawlStats

fn clone(&self) -> CrawlStats

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CrawlStats

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CrawlStats

fn default() -> CrawlStats

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CrawlStats

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for CrawlStats

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.