Crate crw_core 

Core types, configuration, and error handling for the CRW web scraper.

This crate provides the foundational building blocks shared across all CRW crates:

• config — Layered TOML configuration with environment variable overrides
• error — Unified error types (CrwError) and result alias (CrwResult)
• types — Shared data structures (ScrapeData, FetchResult, OutputFormat, etc.)
• url_safety — SSRF protection (blocks private IPs, cloud metadata, non-HTTP schemes)

Example

use crw_core::{AppConfig, CrwError, CrwResult};

let config = AppConfig::load().unwrap();
assert!(config.server.port > 0);

Re-exports

pub use config::AppConfig;

pub use deadline::Deadline;

pub use error::CrwError;

pub use error::CrwResult;

pub use proxy::ProxyEntry;

pub use proxy::ProxyRotation;

pub use proxy::ProxyRotator;

Modules

config

deadline

End-to-end deadline propagation across the scrape pipeline.

error

mcp

Re-export shim. MCP JSON-RPC types now live in the crw-mcp-proto crate so crw-browse can depend on them without pulling in crw-core’s HTTP stack.

metrics

Prometheus metrics for renderer routing, host preferences, and circuit breakers.

proxy

Proxy list + rotation primitives shared across the HTTP, crawl, and CDP paths.

research_types

Firecrawl-Research-API-compatible response types for /v1/search/research/*.

types

url_safety