Expand description
WHATWG byte-stream encoding sniffing for HTML input.
Sniffing precedence per WHATWG § 12.2.3:
- Caller-supplied encoding (HTTP
Content-Type, manual override viaHtmlParseOptions::encoding_override). - Byte-order mark (UTF-8
EF BB BF, UTF-16BEFE FF, UTF-16LEFF FE). - Pre-scan up to
HtmlParseOptions::encoding_sniff_windowbytes (default 1024) for a<meta charset>or<meta http-equiv="Content-Type">declaration. - Fall back to Windows-1252 — explicitly not Latin-1. The HTML5 spec mandates this for backward compatibility with legacy web content.
§What’s not implemented (deferred to v2 if a user asks)
- The full WHATWG prescan algorithm for
<meta>is approximated here. We handle the two common forms (charset attribute and http-equiv content-type) with reasonable attribute parsing, but not every WHATWG edge case (deeply nested comments inside the head, weird whitespace, etc.). - Encoding switches during parsing (the spec allows the parser
to detect a different encoding in a later
<meta>and restart). Our sniffer reads up to the configured window once and then commits to that encoding for the whole document.
Functions§
- decode_
html_ input - Sniff + transcode in one step. Returns UTF-8 bytes plus the
encoding that was used so the caller can record it on the
resulting
Document. - label_
to_ encoding - Map a WHATWG encoding label to our
Encodingenum. Unknown labels fall through toEncoding::Otherwhich routes toencoding_rs(when thefull-encodingsfeature is on) or errors out. - prescan_
meta_ charset - Pre-scan the first chunk of input for a
<meta>tag with a charset declaration. Returns the label string as written, orNoneif no usable declaration is found. - sniff_
html_ encoding - Sniff the encoding of an HTML byte stream per the WHATWG algorithm. Always returns some encoding — falls back to Windows-1252 when no signal is found.