Expand description
HTTP client: rustls, retry, rate limit, body cap, host allowlist, cancel-aware.
§Product fetch posture (Rules Rust — rede + web scraping)
This is a one-shot docs client, not a general crawler or HTTP server:
- GET-only product surface against a fixed HTTPS host allowlist (SSRF gate).
- One primary request per command (no multi-URL frontier, no sitemap/RSS).
- User-Agent identifies
docsrs-cli/{version}plus optional contact. - Politeness: per-host delay floor + additive jitter (
crate::retry::politeness_delay), in-process clock and cross-process lock+stamp. - Body budget: stream with
Content-Lengthearly reject +try_reserve*(no unboundedbytes().await). - TLS: rustls only (
provider=ring, floor ≥0.23.18); min TLS 1.2; webpki-roots; neverdanger_accept_invalid_certs(ADR 0007). - HTTP/2 enabled (feature
http2); ALPN negotiates with the origin. - Proxy: system env
HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXYvia reqwestsystem-proxy(allowlist still applies to the target URL). - Sockets:
TCP_NODELAYon; keepalive idle/interval/retries named constants. - Pool: tiny idle pool sized for one-shot multi-host GETs (not a daemon).
- No cookie jar (
cookie_storefeature not enabled); no shared cookies. - Provenance: HTML payloads expose
source_urlon success envelopes. - robots.txt: PROIBIDO — never fetch/parse/enforce REP (ADR 0003).
- OOS: ETag conditional GETs, anti-bot fingerprinting, headless Chrome,
proxy rotation, DoH/DoT custom resolver, server bind/accept, circuit
breaker, mTLS. See
docs/decisions/0003-web-fetch-scope.md.
§Workload classification (Rules Rust — parallelism + memória)
- Class: mixed — async I/O stage on multi-thread Tokio + CPU parse stage
via
crate::concurrency::ConcurrencyBudget::run_cpu_bound. - Shareable: methods take
&self. Per-host in-process clock is aMutex<HashMap>with sleep outside the lock (noMutexacross.awaitfor the map itself; cross-process flock still owns multi-process throttle). - Budget: each client owns a [
ConcurrencyBudget] forspawn_blockingparse work (and future multi-GET fan-out). Bound is explicit (auto or--max-concurrency). - Network: product commands still issue one primary GET at a time; rate limit + body stream cap remain the I/O backpressure.
§Retry (Rules Rust — retry/backoff)
Policy lives in crate::retry::RetryConfig (not inline magic). GET-only
product surface ⇒ idempotent retries for 408/429/5xx/transport only. Dual
budget: max_attempts + max_elapsed_ms. Retry-After accepts delta-seconds
and HTTP-date. Kill switch --disable-retry / TOML disable_retry /
max_retries=0. See module docs on retry.
Layout (SRP): [client] · [body] · [content_type] · [allowlist] ·
[rate_limit] · [constants].
Structs§
- Http
Client - HTTPS client bound to product allowlist and config.
- Http
Response - Successful HTTP response body and metadata.
Constants§
- POOL_
IDLE_ TIMEOUT_ SECS - Idle connection lifetime in the pool before eviction (seconds).
- POOL_
MAX_ IDLE_ PER_ HOST - Max idle connections retained per host in the reqwest pool (one-shot CLI).
- TCP_
KEEPALIVE_ IDLE_ SECS - TCP keepalive idle probe delay (seconds) — Rules Rust baseline for dead peers.
- TCP_
KEEPALIVE_ INTERVAL_ SECS - TCP keepalive probe interval (seconds).
- TCP_
KEEPALIVE_ RETRIES - TCP keepalive probe retry count before the connection is considered dead.
Functions§
- client_
posture_ detail - Machine-readable summary of the compiled HTTP client posture (doctor / docs).
- content_
type_ looks_ html - True when Content-Type looks like HTML/XHTML.
- content_
type_ looks_ json - True when Content-Type looks like JSON.
- decode_
utf8 - Decode body as UTF-8, stripping a leading UTF-8 BOM.
- require_
content_ type_ html - Fail closed when a Content-Type is present but not HTML/XHTML.
- require_
content_ type_ json - Fail closed when a Content-Type is present but not JSON.