Skip to main content

Crate encoding

Crate encoding 

Source
Expand description

Bytes into Parsec. parse_bytes finds a page’s encoding the way the HTML Standard says a browser does, decodes it and parses it. ByteParser does the same with the bytes as they arrive.

The decoding is encoding_rs, Firefox’s implementation of the WHATWG Encoding Standard, re-exported here. It has every encoding the web uses, decoding and encoding, whole or streamed:

use parsec_encoding::Encoding;

let sjis = Encoding::for_label(b"sjis").unwrap();
assert_eq!(sjis.name(), "Shift_JIS");
let (text, _, had_errors) = sjis.decode(b"\x93\xfa\x96\x7b");
assert_eq!((&*text, had_errors), ("日本", false));
let (bytes, _, _) = sjis.encode("日本");
assert_eq!(&*bytes, b"\x93\xfa\x96\x7b");

Modules§

encoding_rs
encoding_rs is a Gecko-oriented Free Software / Open Source implementation of the Encoding Standard in Rust. Gecko-oriented means that converting to and from UTF-16 is supported in addition to converting to and from UTF-8, that the performance and streamability goals are browser-oriented, and that FFI-friendliness is a goal.

Structs§

ByteParser
Parses a page given as bytes a piece at a time, as a browser does while it downloads one. You get what parse_bytes gives for all the pieces joined: the same document, encoding and confidence, however the bytes were cut.
Encoding
An encoding as defined in the Encoding Standard.
Offsets
Turns offsets into a decoded page into offsets into its bytes, and back, for the spans and parse errors of a document parsed from bytes.
Options
How to turn bytes into a document.
Parsed
A document parsed from bytes, and how its encoding was decided.

Enums§

Confidence
How sure the parser is of an encoding, in the HTML Standard’s terms.

Functions§

locale_default
The fallback encoding a browser uses for a user whose language is locale (a BCP 47 tag such as ja, ru-RU or zh-Hant), from the HTML Standard’s table of suggested defaults. Pass it as Options::fallback when you know your users’ language, or a site’s.
parse_bytes
Parses a page given as bytes, as a browser would: sniffs its encoding (sniff), decodes it and parses it.
prescan
The encoding a <meta> (or an XML declaration) in bytes declares, as the HTML Standard’s prescan finds it, reading bytes and nothing past them. crate::sniff gives it the first 1,024 bytes, as the spec advises; pass more or fewer to scan further or less.
sniff
The HTML Standard’s encoding sniffing algorithm, for a parser that has every byte: the byte order mark, then transport, then the prescan of the first 1,024 bytes, then autodetection, then fallback.