Skip to main content

ByteParser

Struct ByteParser 

Source
pub struct ByteParser { /* private fields */ }
Expand description

Parses a page given as bytes a piece at a time, as a browser does while it downloads one. You get what parse_bytes gives for all the pieces joined: the same document, encoding and confidence, however the bytes were cut.

use parsec_encoding::{ByteParser, Confidence, Options};

let mut parser = ByteParser::new(Options::default());
// Shift_JIS for 日本, cut inside the first character.
for piece in [&b"<meta charset=sjis><p>\x93"[..], b"\xfa\x96\x7b"] {
    parser.feed(piece);
}
let parsed = parser.finish().unwrap();
assert_eq!(parsed.encoding, encoding_rs::SHIFT_JIS);
assert_eq!(parsed.confidence, Confidence::Certain);
assert_eq!(parsed.document.text(parsed.document.root()), "日本");

How it goes, in the HTML Standard’s terms:

  • Until it knows the encoding, it holds the bytes back. A byte order mark decides at once, even cut over two pieces. Then the transport layer’s encoding, if you gave one. Otherwise the prescan does, which the spec lets a browser wait up to 1,024 bytes for: this waits until the prescan finds a whole <meta> naming an encoding, or there are 1,024 bytes, or the input ends. With no <meta>, the bytes held back go through the autodetection (sniff): non-ASCII and valid UTF-8, with a character cut off at the end allowed, is UTF-8; otherwise it’s the fallback.
  • From then on each piece goes through encoding_rs’s Decoder, which keeps a character cut between pieces for the next one, and on into parsec::Parser.
  • When the encoding was only a guess, a <meta> the parser meets later can name another. The spec has the browser start again in that one. Here that’s at ByteParser::finish: one more decode and parse, of every byte, once at most, as parse_bytes does. So is a guess the whole input wouldn’t make (UTF-8 that starts after 1,024 bytes of ASCII, or bytes after the first 1,024 that aren’t UTF-8): with every byte kept, finish checks and, if so, parses as parse_bytes would.

Memory: while the encoding is a guess (no byte order mark, no transport encoding), every byte is kept for that second parse, so the input sits in memory once as bytes beside the document, which holds the decoded text anyway. Give the transport layer’s encoding when you have it and only the decoder’s few bytes of state are kept. Before the encoding is known, the pieces fed so far wait, which stops once there are 1,024 bytes.

A time budget in Options::html counts the time spent in ByteParser::feed and ByteParser::finish, both parses together, not the time waiting for input.

Implementations§

Source§

impl ByteParser

Source

pub fn new(options: Options) -> ByteParser

Source

pub fn feed(&mut self, chunk: &[u8])

The next piece of the input.

Source

pub fn finish(self) -> Result<Parsed, ParseError>

The end of the input: the document and its encoding, or why there’s none, as parse_bytes gives them.

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.