Skip to main content

Module http

Module http 

Source
Expand description

HTTP client: rustls, retry, rate limit, body cap, host allowlist, cancel-aware.

§Product fetch posture (Rules Rust — web scraping / crawling)

This is a one-shot docs client, not a general crawler:

  • GET-only product surface against a fixed HTTPS host allowlist.
  • One primary request per command (no multi-URL frontier, no sitemap/RSS).
  • User-Agent identifies docsrs-cli/{version} plus optional contact.
  • Politeness: per-host delay floor + additive jitter (crate::retry::politeness_delay), in-process clock and cross-process lock+stamp.
  • Body budget: stream with Content-Length early reject + try_reserve* (no unbounded bytes().await).
  • TLS: rustls only; never danger_accept_invalid_certs.
  • No cookie jar (cookie_store feature not enabled); no shared cookies.
  • Provenance: HTML payloads expose source_url on success envelopes.
  • OOS: robots.txt REP parser, meta robots / X-Robots-Tag, ETag conditional GETs, anti-bot TLS fingerprinting, headless Chrome, proxy rotation, MinHash. See docs/decisions/0003-web-fetch-scope.md.

§Workload classification (Rules Rust — parallelism)

  • Class: mixed — async I/O stage on multi-thread Tokio + CPU parse stage via crate::concurrency::ConcurrencyBudget::run_cpu_bound.
  • Shareable: methods take &self. Per-host in-process clock is a Mutex<HashMap> with sleep outside the lock (no Mutex across .await for the map itself; cross-process flock still owns multi-process throttle).
  • Budget: each client owns a ConcurrencyBudget for spawn_blocking parse work (and future multi-GET fan-out). Bound is explicit (auto or --max-concurrency).
  • Network: product commands still issue one primary GET at a time; rate limit + body stream cap remain the I/O backpressure.

§Retry (Rules Rust — retry/backoff)

Policy lives in crate::retry::RetryConfig (not inline magic). GET-only product surface ⇒ idempotent retries for 429/5xx/transport only. Kill switch --disable-retry / DOCSRS_CLI_DISABLE_RETRY. See module docs on retry.

Structs§

HttpClient
HTTPS client bound to product allowlist and config.
HttpResponse
Successful HTTP response body and metadata.

Functions§

content_type_looks_html
True when Content-Type looks like HTML/XHTML.
content_type_looks_json
True when Content-Type looks like JSON.
decode_utf8
Decode body as UTF-8, stripping a leading UTF-8 BOM.