Expand description
HTTP primitives shared by the fetch and search paths: the user agent, the response-body cap, and the retry classification. Both paths previously carried their own copy of the last two and had drifted apart — search read bodies without any cap at all.
Nothing here touches a concrete HTTP client; the transport-level pieces live
in crate::transport.
Constants§
- ERROR_
BODY_ CAP - Cap on a 4xx or 5xx body (64 KiB). Error bodies only feed a message, and the provider’s own explanation fits well inside this.
- MAX_
BODY_ BYTES - Hard cap on the response body we will read (5 MiB). The HTML extractor turns a page into a few KB of text, so a multi-megabyte body is almost never worth the bandwidth, memory, and parse time — and an unbounded read is a DoS lever. Bodies over the cap are truncated (not errored): partial content is still useful and the extractor copes with truncated HTML.
- USER_
AGENT - The user agent both paths send.
Functions§
- status_
text - A status code and its reason phrase, as
404 Not Found. A code with no standard phrase is shown as the number alone. - transient_
status - Is a response status worth retrying? Server errors and explicit throttling.