Skip to main content

Module encoding

Module encoding 

Source
Expand description

WHATWG byte-stream encoding sniffing for HTML input.

Sniffing precedence per WHATWG § 12.2.3:

  1. Caller-supplied encoding (HTTP Content-Type, manual override via HtmlParseOptions::encoding_override).
  2. Byte-order mark (UTF-8 EF BB BF, UTF-16BE FE FF, UTF-16LE FF FE).
  3. Pre-scan up to HtmlParseOptions::encoding_sniff_window bytes (default 1024) for a <meta charset> or <meta http-equiv="Content-Type"> declaration.
  4. Fall back to Windows-1252 — explicitly not Latin-1. The HTML5 spec mandates this for backward compatibility with legacy web content.

§What’s not implemented (deferred to v2 if a user asks)

  • The full WHATWG prescan algorithm for <meta> is approximated here. We handle the two common forms (charset attribute and http-equiv content-type) with reasonable attribute parsing, but not every WHATWG edge case (deeply nested comments inside the head, weird whitespace, etc.).
  • Encoding switches during parsing (the spec allows the parser to detect a different encoding in a later <meta> and restart). Our sniffer reads up to the configured window once and then commits to that encoding for the whole document.

Functions§

decode_html_input
Sniff + transcode in one step. Returns UTF-8 bytes plus the encoding that was used so the caller can record it on the resulting Document.
label_to_encoding
Map a WHATWG encoding label to our Encoding enum. Unknown labels fall through to Encoding::Other which routes to encoding_rs (when the full-encodings feature is on) or errors out.
prescan_meta_charset
Pre-scan the first chunk of input for a <meta> tag with a charset declaration. Returns the label string as written, or None if no usable declaration is found.
sniff_html_encoding
Sniff the encoding of an HTML byte stream per the WHATWG algorithm. Always returns some encoding — falls back to Windows-1252 when no signal is found.