Struct StreamTokenizer 

pub struct StreamTokenizer { /* private fields */ }

A streaming tokenizer that processes input chunk by chunk.

Maintains internal state so that token boundaries that span chunk boundaries are handled correctly.

Example

use fhp_tokenizer::streaming::StreamTokenizer;
use fhp_tokenizer::token::Token;

let mut tokenizer = StreamTokenizer::new();
let mut all_tokens: Vec<Token<'static>> = Vec::new();

let html = b"<div>hello</div>";
// Feed in small chunks.
let owned = tokenizer.feed(&html[..5]);
all_tokens.extend(owned);
let owned = tokenizer.feed(&html[5..]);
all_tokens.extend(owned);
let owned = tokenizer.finish();
all_tokens.extend(owned);

assert!(all_tokens.iter().any(|t| matches!(t, Token::OpenTag { .. })));

Implementations

impl StreamTokenizer

pub fn new() -> Self

Create a new streaming tokenizer.

pub fn buffered_len(&self) -> usize

Number of bytes currently buffered as residual (not yet processed).

Bounded by MAX_RAW_TEXT_RESIDUAL plus the size of one fed chunk.

pub fn feed(&mut self, chunk: &[u8]) -> Vec<Token<'static>>

Feed a chunk of UTF-8 input and return any complete tokens.

Tokens are returned as owned ('static lifetime) since the chunk data may not live long enough. Text content is cloned into Cow::Owned.

pub fn feed_str_with(&mut self, chunk: &str, on_token: impl FnMut(&Token<'_>))

Feed a UTF-8 chunk and process complete tokens via callback without cloning.

This path is intended for internal high-throughput consumers (e.g. tree building) that can consume tokens immediately.

pub fn finish(&mut self) -> Vec<Token<'static>>

Signal end of input and flush any remaining buffered data.

pub fn finish_with(&mut self, on_token: impl FnMut(&Token<'_>))

Signal end of input and flush buffered tokens via callback without cloning.

Trait Implementations

impl Default for StreamTokenizer

fn default() -> Self

Returns the “default value” for a type.