pub struct Tokenizer { /* private fields */ }Expand description
A stateful tokenizer for one root TextMate grammar.
Implementations§
Source§impl Tokenizer
impl Tokenizer
Sourcepub fn new(
registry: &GrammarRegistry,
root: GrammarId,
options: TokenizerOptions,
) -> Result<Self>
pub fn new( registry: &GrammarRegistry, root: GrammarId, options: TokenizerOptions, ) -> Result<Self>
Creates a tokenizer for a root grammar owned by registry.
Sourcepub fn for_bundled_language(
language: &str,
options: TokenizerOptions,
) -> Result<Self>
Available on crate feature bundled-grammars only.
pub fn for_bundled_language( language: &str, options: TokenizerOptions, ) -> Result<Self>
bundled-grammars only.Constructs a tokenizer from the bundled grammar catalog.
Sourcepub fn from_prepared(
prepared: &PreparedLanguage,
options: TokenizerOptions,
) -> Self
pub fn from_prepared( prepared: &PreparedLanguage, options: TokenizerOptions, ) -> Self
Constructs a tokenizer from caller-owned immutable preparation.
Sourcepub fn initial_state(&self) -> TokenizerState
pub fn initial_state(&self) -> TokenizerState
Creates initial continuation state owned by this tokenizer.
The returned state is document start: \A may match on the first
subsequent line. Later continuation states do not rematch \A.
Sourcepub fn tokenize_line(
&mut self,
line: &str,
state: &mut TokenizerState,
) -> Result<TokenizedLine>
pub fn tokenize_line( &mut self, line: &str, state: &mut TokenizerState, ) -> Result<TokenizedLine>
Tokenizes one logical line. line must not include a newline terminator.
\A matches only while state is this tokenizer’s initial
document-start state, matching Tokenizer::tokenize. After a
successful call, state is no longer at document start.
Sourcepub fn tokenize_line_into(
&mut self,
line: &str,
state: &mut TokenizerState,
tokens: &mut Vec<Token>,
) -> Result<HighlightStatus>
pub fn tokenize_line_into( &mut self, line: &str, state: &mut TokenizerState, tokens: &mut Vec<Token>, ) -> Result<HighlightStatus>
Tokenizes one logical line into a caller-owned reusable buffer.
The buffer is cleared after input validation and retains its capacity
for subsequent calls. This is the allocation-conscious counterpart to
Tokenizer::tokenize_line.
Sourcepub fn tokenize_line_with(
&mut self,
line: &str,
state: &mut TokenizerState,
sink: impl FnMut(Token),
) -> Result<HighlightStatus>
pub fn tokenize_line_with( &mut self, line: &str, state: &mut TokenizerState, sink: impl FnMut(Token), ) -> Result<HighlightStatus>
Tokenizes one logical line and sends each token to sink in byte order.
The callback receives owned tokens backed by shared immutable scope names, so no output collection is required. It is not called when input validation fails. The returned status covers the complete line.
Sourcepub fn tokenize(&mut self, source: &str) -> TokenizedDocument
pub fn tokenize(&mut self, source: &str) -> TokenizedDocument
Tokenizes a complete UTF-8 source document.
Sourcepub fn checkpoints(&self, interval: usize) -> CheckpointTable
pub fn checkpoints(&self, interval: usize) -> CheckpointTable
Creates an owned checkpoint table for viewport tokenization.
Sourcepub fn tokenize_viewport(
&mut self,
source: &str,
visible_lines: Range<usize>,
checkpoints: &mut CheckpointTable,
) -> Result<TokenizedDocument>
pub fn tokenize_viewport( &mut self, source: &str, visible_lines: Range<usize>, checkpoints: &mut CheckpointTable, ) -> Result<TokenizedDocument>
Tokenizes a line viewport while replaying from the nearest checkpoint.