Crate crawlex 

Re-exports

pub use config::Config;

pub use config::ConfigBuilder;

pub use config::RenderMode;

pub use crawler::Crawler;

pub use error::Error;

pub use error::Result;

pub use hooks::HookDecision;

pub use hooks::HookEvent;

pub use status::Status;

pub use status::TerminalReason;

Modules

adblock

Ad/tracker URL gate (slice 20).

antibot

Phase 1 antibot/stealth module — rich vendor detection, per-session ChallengeState, and pure detect_from_* functions.

cache_validator

cli

config

crawl_stats

crawler

discovery

error

Error taxonomy for crawlex.

events

NDJSON event bus — public contract between the crawler runtime and any external consumer (CLI piping, SDKs, hooks).

extract

Content extraction and link filtering.

fallback_fetch

fingerprint

Fingerprinting (fingerprint/).

from_curl

crawlex from-curl '<curl …>' — Slice 21.

frontier

hooks

Lifecycle hooks — pluggable callbacks fired at well-known points in the crawl pipeline.

http

HTTP-layer helpers owned by the crawl-pattern scheduler (wave 1).

identity

IdentityBundle — coherent “who is the browser pretending to be?” record.

impersonate

mcp

Slice 24 — crawlex mcp JSON-RPC 2.0 server over stdio.

metrics

Performance KPIs — network timings for the HTTP path, Core Web Vitals and runtime counters for the render path. Both populated per-page; stored in page_metrics and exposed to hooks via HookContext.user_data["metrics"].

metrics_server

Minimal Prometheus scrape endpoint using a hand-rolled HTTP/1.0 responder.

parser

pattern

URL match patterns: globs that compile to anchored regexes.

policy

Policy Engine — deterministic, explainable crawl decisions.

proxy

queue

render

Render/browser path. Only compiled when cdp-backend is enabled. crawlex-mini builds without this module; callers that need runtime “render not available” errors use Error::RenderDisabled.

robots

runner

Runner module — per-Job execution seam.

scheduler

Render scheduler with per-host / per-origin / per-proxy / per-session inflight budgets. Sits in front of RenderPool so a single noisy origin can’t monopolise the browser or trigger rate limits upstream.

scraping

v2 scraping framework — top-level entry points.

script

ScriptSpec v1 — unified AST for declarative crawl scripts.

shell

status

Canonical status taxonomy (slice 1).

storage

url_util

wait_strategy

Page-load wait strategy.