Module http 

HTTP primitives shared by the fetch and search paths: the user agent, the response-body cap, and the retry classification. Both paths previously carried their own copy of the last two and had drifted apart — search read bodies without any cap at all.

Constants

MAX_BODY_BYTES

Hard cap on the response body we will read (5 MiB). The HTML extractor turns a page into a few KB of text, so a multi-megabyte body is almost never worth the bandwidth, memory, and parse time — and an unbounded read is a DoS lever. Bodies over the cap are truncated (not errored): partial content is still useful and the extractor copes with truncated HTML.

USER_AGENT

The user agent both paths send.

Functions

read_body_capped

Read a response body, streaming chunks with a running byte cap so an oversized body is bounded before it is ever parsed. The bool in the error reports whether the read failure is transient (worth retrying).

read_body_capped_bytes

read_body_capped without the decoding step, for callers that need to apply the response’s declared charset themselves.

transient_send_error

Is a send/connect failure worth retrying?

transient_status

Is a response status worth retrying? Server errors and explicit throttling.