Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Available on crate feature html only.

A byte-faithful, low-allocation, resumable HTML tokenizer.

Implementations

impl Tokenizer

pub fn new() -> Self

Creates a new tokenizer.

Non-strict by default (rama’s convention): ambiguous text-mode contexts are tokenized best-effort. Enable with_strict to abort on such ambiguity instead.

pub fn with_strict(self, strict: bool) -> Self

Sets strict mode. When enabled, ambiguous text-mode contexts abort with a ParsingAmbiguityError instead of tokenizing best-effort.

pub fn set_strict(&mut self, strict: bool) -> &mut Self

Sets strict mode. When enabled, ambiguous text-mode contexts abort with a ParsingAmbiguityError instead of tokenizing best-effort.

pub fn write<S: TokenSink>( &mut self, chunk: &[u8], sink: &mut S, ) -> Result<(), ParsingAmbiguityError>

Feeds a chunk of input, emitting every token that is now complete.

Errors

See tokenize.

pub fn end<S: TokenSink>( &mut self, sink: &mut S, ) -> Result<(), ParsingAmbiguityError>

Finalizes the stream, emitting any remaining (possibly unterminated) tokens, then resets for reuse.

Errors

See tokenize.

pub fn tokenize<S: TokenSink>( &mut self, input: &[u8], sink: &mut S, ) -> Result<(), ParsingAmbiguityError>

Tokenizes a complete input in one shot (write then end).

Errors

See tokenize.

Trait Implementations

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.