Crate spider_util 

spider-util

Shared types and helper modules used across the spider-* workspace.

This crate is where request and response models, error types, selector helpers, formatting helpers, metrics helpers, and other common utilities live.

Example

use spider_util::request::Request;
use url::Url;

let url = Url::parse("https://example.com").unwrap();
let request = Request::new(url);
let _ = request;

Re-exports

pub use formatters::ByteFormatter;

pub use formatters::DefaultByteFormatter;

pub use formatters::DefaultDurationFormatter;

pub use formatters::DefaultRateCalculator;

pub use formatters::DurationFormatter;

pub use formatters::RateCalculator;

pub use formatters::format_bytes;

pub use formatters::format_duration;

pub use formatters::format_request_time;

pub use http_client::HttpClient;

pub use constants::*;

Modules

bloom

Bloom filter used by the scheduler for cheap duplicate checks.

constants

Workspace-wide public constants.

error

Shared error types for the workspace.

formatters

Formatting helpers for metrics and runtime output.

http_client

Small HTTP client abstraction used by middleware.

item

Item traits and parse results.

metrics

Metrics helpers shared by runtime reporting code.

request

Request types used by the crawler runtime.

response

Response types and response-side helpers.

selector

Cached CSS selector helpers.

util

Small utility helpers shared across the workspace.