Expand description
The fetch layer: impersonated HTTP, politeness, and a headless fallback.
Three things separate this from a bare reqwest loop:
- Fingerprint. Cloudflare and friends read the TLS ClientHello (JA3/JA4)
and the HTTP/2 SETTINGS frame, not just the User-Agent.
wreqreplays a real Chrome’s, so a plainGETgets a plain200where a stock client gets a challenge page. - Politeness.
robots.txt, per-host spacing and a concurrency ceiling are on by default, because a fast crawler with none of those is a way to get a user’s IP banned. - Escalation. JS-gated pages are detected from the response and, when
the
browserfeature is on, retried through headless Chrome — which is two orders of magnitude more expensive and therefore never the first move.
Modules§
- robots
- A small, strict
robots.txtreader.
Structs§
- Fetch
Config - Everything tunable about fetching.
- Fetcher
- A pooled, polite, fingerprint-aware HTTP client.
- Page
- A fetched document.
Enums§
- Impersonate
- How the client should present itself on the wire.
Constants§
- DEFAULT_
USER_ AGENT - Default identity when impersonation is off, or for the
robots.txtlookup. - ROBOTS_
AGENT - The token we match
robots.txtgroups against.
Functions§
- needs_
javascript - Does this response look like an empty shell waiting for JavaScript?
- normalize
- Reject anything that is not an absolute http(s) URL.