Skip to main content

sniff

Function sniff 

Source
pub fn sniff(
    bytes: &[u8],
    transport: Option<&'static Encoding>,
    fallback: &'static Encoding,
) -> (&'static Encoding, Confidence)
Expand description

The HTML Standard’s encoding sniffing algorithm, for a parser that has every byte: the byte order mark, then transport, then the prescan of the first 1,024 bytes, then autodetection, then fallback.

The spec lets a browser guess from the bytes before falling back, and browsers do: Firefox’s chardetng and Chrome’s CED both recognise UTF-8. Parsec’s guess is that one only, and deterministic: bytes with something beyond ASCII that are all valid UTF-8 are UTF-8, tentative, so a later <meta> can still change it. Anything else falls back.

use parsec_encoding::{Confidence, sniff};
let page = b"<meta charset=windows-1251><p>\xcf\xf0\xe8\xe2\xe5\xf2";
let found = sniff(page, None, encoding_rs::WINDOWS_1252);
assert_eq!(found, (encoding_rs::WINDOWS_1251, Confidence::Tentative));