Struct LineTokenizer 

pub struct LineTokenizer<'grammar> { /* private fields */ }

Reusable line tokenizer for one compiled grammar.

The tokenizer keeps runtime scratch that is not part of immutable grammar data, such as dynamically resolved end regexes. Reuse one tokenizer across a blob or visible range to avoid recompiling those regexes unnecessarily.

Implementations

impl<'grammar> LineTokenizer<'grammar>

pub fn new(grammar: &'grammar Grammar) -> Self

Creates a tokenizer for a compiled grammar.

pub fn clear_caches(&mut self)

Clears cached dynamic end regexes while retaining allocations.

pub fn tokenize_line( &mut self, state: &mut LineState, line: &str, ) -> Vec<ScopeSpan>

Tokenizes one line, mutating the caller-owned line state.

pub fn tokenize_line_into( &mut self, state: &mut LineState, line: &str, spans: &mut Vec<ScopeSpan>, )

Tokenizes one line into a caller-owned buffer.

Trait Implementations

impl<'grammar> Debug for LineTokenizer<'grammar>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.