Skip to main content

Module http

Module http 

Source
Expand description

HTTP client: rustls, retry, rate limit, body cap, host allowlist, cancel-aware.

§Product fetch posture (Rules Rust — rede + web scraping)

This is a one-shot docs client, not a general crawler or HTTP server:

  • GET-only product surface against a fixed HTTPS host allowlist (SSRF gate).
  • One primary request per command (no multi-URL frontier, no sitemap/RSS).
  • User-Agent identifies docsrs-cli/{version} plus optional contact.
  • Politeness: per-host delay floor + additive jitter (crate::retry::politeness_delay), in-process clock and cross-process lock+stamp.
  • Body budget: stream with Content-Length early reject + try_reserve* (no unbounded bytes().await).
  • TLS: rustls only (provider=ring, floor ≥0.23.18); min TLS 1.2; webpki-roots; never danger_accept_invalid_certs (ADR 0007).
  • HTTP/2 enabled (feature http2); ALPN negotiates with the origin.
  • Proxy: system env HTTP_PROXY / HTTPS_PROXY / ALL_PROXY / NO_PROXY via reqwest system-proxy (allowlist still applies to the target URL).
  • Sockets: TCP_NODELAY on; keepalive idle/interval/retries named constants.
  • Pool: tiny idle pool sized for one-shot multi-host GETs (not a daemon).
  • No cookie jar (cookie_store feature not enabled); no shared cookies.
  • Provenance: HTML payloads expose source_url on success envelopes.
  • robots.txt: PROIBIDO — never fetch/parse/enforce REP (ADR 0003).
  • OOS: ETag conditional GETs, anti-bot fingerprinting, headless Chrome, proxy rotation, DoH/DoT custom resolver, server bind/accept, circuit breaker, mTLS. See docs/decisions/0003-web-fetch-scope.md.

§Workload classification (Rules Rust — parallelism + memória)

  • Class: mixed — async I/O stage on multi-thread Tokio + CPU parse stage via crate::concurrency::ConcurrencyBudget::run_cpu_bound.
  • Shareable: methods take &self. Per-host in-process clock is a Mutex<HashMap> with sleep outside the lock (no Mutex across .await for the map itself; cross-process flock still owns multi-process throttle).
  • Budget: each client owns a [ConcurrencyBudget] for spawn_blocking parse work (and future multi-GET fan-out). Bound is explicit (auto or --max-concurrency).
  • Network: product commands still issue one primary GET at a time; rate limit + body stream cap remain the I/O backpressure.

§Retry (Rules Rust — retry/backoff)

Policy lives in crate::retry::RetryConfig (not inline magic). GET-only product surface ⇒ idempotent retries for 408/429/5xx/transport only. Dual budget: max_attempts + max_elapsed_ms. Retry-After accepts delta-seconds and HTTP-date. Kill switch --disable-retry / TOML disable_retry / max_retries=0. See module docs on retry.

Layout (SRP): [client] · [body] · [content_type] · [allowlist] · [rate_limit] · [constants].

Structs§

HttpClient
HTTPS client bound to product allowlist and config.
HttpResponse
Successful HTTP response body and metadata.

Constants§

POOL_IDLE_TIMEOUT_SECS
Idle connection lifetime in the pool before eviction (seconds).
POOL_MAX_IDLE_PER_HOST
Max idle connections retained per host in the reqwest pool (one-shot CLI).
TCP_KEEPALIVE_IDLE_SECS
TCP keepalive idle probe delay (seconds) — Rules Rust baseline for dead peers.
TCP_KEEPALIVE_INTERVAL_SECS
TCP keepalive probe interval (seconds).
TCP_KEEPALIVE_RETRIES
TCP keepalive probe retry count before the connection is considered dead.

Functions§

client_posture_detail
Machine-readable summary of the compiled HTTP client posture (doctor / docs).
content_type_looks_html
True when Content-Type looks like HTML/XHTML.
content_type_looks_json
True when Content-Type looks like JSON.
decode_utf8
Decode body as UTF-8, stripping a leading UTF-8 BOM.
require_content_type_html
Fail closed when a Content-Type is present but not HTML/XHTML.
require_content_type_json
Fail closed when a Content-Type is present but not JSON.