Struct LexerState 

pub struct LexerState<S, L: Language> { /* private fields */ }

State information for incremental lexical analysis.

This struct maintains the current position and context during tokenization, enabling incremental and resumable lexing operations.

Implementations

impl<S: Source, L: Language> LexerState<S, L>

pub fn new(source: S) -> Self

Creates a new lexer state with the given source text.

Arguments

• source - The source text to lex

Returns

A new LexerState initialized at the beginning of the source

pub fn new_with_cache( source: S, changed: usize, cache: IncrementalCache<'_, L>, ) -> Self

Creates a new lexer state with the given source text and incremental cache.

Arguments

• source - The source text to lex
• changed - The number of bytes that have changed since the last lex
• cache - The incremental cache containing previous lexing results

Returns

A new LexerState initialized at the beginning of the source with cache support

pub fn rest(&self) -> &str

Gets the remaining text from the current position to the end of the source.

Returns

A string slice containing the remaining text

pub fn get_position(&self) -> usize

Gets the current byte offset position in the source text.

Returns

The current byte offset from the start of the source text

pub fn set_position(&mut self, offset: usize) -> usize

Sets the current position to the specified byte offset.

Arguments

• offset - The new byte offset position

Returns

The previous byte offset position

pub fn get_length(&self) -> usize

Gets the total length of the source text in bytes.

Returns

The total number of bytes in the source text

pub fn add_error(&mut self, error: impl Into<OakError>)

Adds an error to the lexer state.

Arguments

• error - The error to add to the diagnostics

pub fn add_token(&mut self, kind: L::SyntaxKind, start: usize, end: usize)

Adds a token to the lexer state.

Arguments

• kind - The kind of the token
• start - The starting byte offset of the token
• end - The ending byte offset of the token

pub fn current(&self) -> Option<char>

Gets the current character at the current position.

Returns

The current character, or None if at the end of the source

pub fn peek(&self) -> Option<char>

Peeks at the next character without advancing the position.

Returns

The next character, or None if at the end of the source

pub fn peek_next_n(&self, n: usize) -> Option<char>

Peeks at the character n positions ahead without advancing the position.

Arguments

• n - The number of characters to peek ahead

Returns

The character n positions ahead, or None if beyond the end of the source

pub fn advance(&mut self, length: usize) -> usize

Advances the position by the specified number of bytes.

Arguments

• length - The number of bytes to advance

Returns

The new byte offset position

pub fn advance_with(&mut self, token: Token<L::SyntaxKind>) -> usize

Advances the position by the specified number of bytes and adds a token.

Arguments

• length - The number of bytes to advance
• token - The kind of token to add

Returns

The new byte offset position

Note

The caller must ensure that the advance is at character boundaries.

pub fn take_while(&mut self, pred: impl FnMut(char) -> bool) -> Range<usize>

Consumes characters while the predicate returns true, returning the consumed range.

Arguments

• pred - The predicate function that determines whether to consume a character

Returns

The byte range of consumed characters

pub fn not_at_end(&self) -> bool

Checks if the lexer has not reached the end of the source text.

Returns

true if not at the end of the source, false otherwise

pub fn safe_check(&mut self, safe_point: usize)

Performs a safety check to prevent infinite loops during lexing.

This method ensures that the lexer always makes progress by forcing advancement when stuck at the same position. It’s used as a safeguard against infinite loops in lexer implementations.

Arguments

• safe_point - The position to check against for potential deadlock

pub fn finish(self, result: Result<(), OakError>) -> LexOutput<L>

Finishes lexing and returns the final output with tokens and diagnostics.

Returns

A LexOutput containing the collected tokens and any errors encountered

Trait Implementations

impl<S: Debug, L: Debug + Language> Debug for LexerState<S, L>

where L::SyntaxKind: Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<S: Source, L: Language> Deref for LexerState<S, L>

type Target = S

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.