pub struct ByteParser { /* private fields */ }Expand description
Parses a page given as bytes a piece at a time, as a browser does while
it downloads one. You get what parse_bytes
gives for all the pieces joined: the same document, encoding and
confidence, however the bytes were cut.
use parsec_encoding::{ByteParser, Confidence, Options};
let mut parser = ByteParser::new(Options::default());
// Shift_JIS for 日本, cut inside the first character.
for piece in [&b"<meta charset=sjis><p>\x93"[..], b"\xfa\x96\x7b"] {
parser.feed(piece);
}
let parsed = parser.finish().unwrap();
assert_eq!(parsed.encoding, encoding_rs::SHIFT_JIS);
assert_eq!(parsed.confidence, Confidence::Certain);
assert_eq!(parsed.document.text(parsed.document.root()), "日本");How it goes, in the HTML Standard’s terms:
- Until it knows the encoding, it holds the bytes back. A byte order
mark decides at once, even cut over two pieces. Then the transport
layer’s encoding, if you gave one. Otherwise the prescan does, which
the spec lets a browser wait up to 1,024 bytes for: this waits until
the prescan finds a whole
<meta>naming an encoding, or there are 1,024 bytes, or the input ends. With no<meta>, the bytes held back go through the autodetection (sniff): non-ASCII and valid UTF-8, with a character cut off at the end allowed, is UTF-8; otherwise it’s the fallback. - From then on each piece goes through encoding_rs’s
Decoder, which keeps a character cut between pieces for the next one, and on intoparsec::Parser. - When the encoding was only a guess, a
<meta>the parser meets later can name another. The spec has the browser start again in that one. Here that’s atByteParser::finish: one more decode and parse, of every byte, once at most, asparse_bytesdoes. So is a guess the whole input wouldn’t make (UTF-8 that starts after 1,024 bytes of ASCII, or bytes after the first 1,024 that aren’t UTF-8): with every byte kept,finishchecks and, if so, parses asparse_byteswould.
Memory: while the encoding is a guess (no byte order mark, no transport encoding), every byte is kept for that second parse, so the input sits in memory once as bytes beside the document, which holds the decoded text anyway. Give the transport layer’s encoding when you have it and only the decoder’s few bytes of state are kept. Before the encoding is known, the pieces fed so far wait, which stops once there are 1,024 bytes.
A time budget in Options::html counts the time spent in
ByteParser::feed and ByteParser::finish, both parses together,
not the time waiting for input.
Implementations§
Source§impl ByteParser
impl ByteParser
pub fn new(options: Options) -> ByteParser
Sourcepub fn finish(self) -> Result<Parsed, ParseError>
pub fn finish(self) -> Result<Parsed, ParseError>
The end of the input: the document and its encoding, or why there’s
none, as parse_bytes gives them.