Expand description
HTTP client: rustls, retry, rate limit, body cap, host allowlist, cancel-aware.
§Product fetch posture (Rules Rust — web scraping / crawling)
This is a one-shot docs client, not a general crawler:
- GET-only product surface against a fixed HTTPS host allowlist.
- One primary request per command (no multi-URL frontier, no sitemap/RSS).
- User-Agent identifies
docsrs-cli/{version}plus optional contact. - Politeness: per-host delay floor + additive jitter (
crate::retry::politeness_delay), in-process clock and cross-process lock+stamp. - Body budget: stream with
Content-Lengthearly reject +try_reserve*(no unboundedbytes().await). - TLS: rustls only; never
danger_accept_invalid_certs. - No cookie jar (
cookie_storefeature not enabled); no shared cookies. - Provenance: HTML payloads expose
source_urlon success envelopes. - OOS: robots.txt REP parser, meta robots / X-Robots-Tag, ETag conditional
GETs, anti-bot TLS fingerprinting, headless Chrome, proxy rotation, MinHash.
See
docs/decisions/0003-web-fetch-scope.md.
§Workload classification (Rules Rust — parallelism)
- Class: mixed — async I/O stage on multi-thread Tokio + CPU parse stage
via
crate::concurrency::ConcurrencyBudget::run_cpu_bound. - Shareable: methods take
&self. Per-host in-process clock is aMutex<HashMap>with sleep outside the lock (noMutexacross.awaitfor the map itself; cross-process flock still owns multi-process throttle). - Budget: each client owns a
ConcurrencyBudgetforspawn_blockingparse work (and future multi-GET fan-out). Bound is explicit (auto or--max-concurrency). - Network: product commands still issue one primary GET at a time; rate limit + body stream cap remain the I/O backpressure.
§Retry (Rules Rust — retry/backoff)
Policy lives in crate::retry::RetryConfig (not inline magic). GET-only
product surface ⇒ idempotent retries for 429/5xx/transport only. Kill switch
--disable-retry / DOCSRS_CLI_DISABLE_RETRY. See module docs on retry.
Structs§
- Http
Client - HTTPS client bound to product allowlist and config.
- Http
Response - Successful HTTP response body and metadata.
Functions§
- content_
type_ looks_ html - True when Content-Type looks like HTML/XHTML.
- content_
type_ looks_ json - True when Content-Type looks like JSON.
- decode_
utf8 - Decode body as UTF-8, stripping a leading UTF-8 BOM.