Skip to main content

Module http

Module http 

Source
Expand description

The fetch layer: impersonated HTTP, politeness, and a headless fallback.

Three things separate this from a bare reqwest loop:

  • Fingerprint. Cloudflare and friends read the TLS ClientHello (JA3/JA4) and the HTTP/2 SETTINGS frame, not just the User-Agent. wreq replays a real Chrome’s, so a plain GET gets a plain 200 where a stock client gets a challenge page.
  • Politeness. robots.txt, per-host spacing and a concurrency ceiling are on by default, because a fast crawler with none of those is a way to get a user’s IP banned.
  • Escalation. JS-gated pages are detected from the response and, when the browser feature is on, retried through headless Chrome — which is two orders of magnitude more expensive and therefore never the first move.

Modules§

robots
A small, strict robots.txt reader.

Structs§

FetchConfig
Everything tunable about fetching.
Fetcher
A pooled, polite, fingerprint-aware HTTP client.
Page
A fetched document.

Enums§

Impersonate
How the client should present itself on the wire.

Constants§

DEFAULT_USER_AGENT
Default identity when impersonation is off, or for the robots.txt lookup.
ROBOTS_AGENT
The token we match robots.txt groups against.

Functions§

needs_javascript
Does this response look like an empty shell waiting for JavaScript?
normalize
Reject anything that is not an absolute http(s) URL.