Struct Lexer 

pub struct Lexer<'a> { /* private fields */ }

Implementations

impl<'a> Lexer<'a>

pub fn new(src: &'a str, options: Options) -> Self

pub fn source(&self) -> &str

Full immutable source supplied to this lexer.

pub fn remaining(&self) -> &str

Source remaining at the live cursor.

pub fn forward(&self, max_chars: usize) -> &str

Return at most max_chars Unicode scalar values from the live cursor. This is the Rust counterpart of the public lex.fwd/Lex.Fwd helper.

pub fn point(&self) -> Point

Snapshot the live cursor for token construction.

pub fn advance_chars(&mut self, count: usize) -> bool

Advance by Unicode scalar values. Returns false without moving when the requested count extends beyond end-of-source.

pub fn token( &self, name: impl AsRef<str>, tin: Tin, value: Value, source: impl Into<TokenText>, point: Point, ) -> Token

Construct a token from a point captured before cursor advancement.

pub fn token_tin(&mut self, name: impl Into<String>) -> Tin

Resolve or allocate a token identity in this lexer’s configuration.

pub fn token_name(&self, tin: Tin) -> String

Resolve a token identity back to its configured name.

pub fn alt_enabled(&self, alt: &AltSpec) -> bool

Whether the options this lexer runs under enable alt: not when rule.exclude names one of its groups, nor when rule.include lists groups and it declares none of them. The parser skips such an alternate, and TypeScript removes it from the rule spec (filterRules) before anything reads the spec, so a custom matcher that reads a rule’s alternates, to tell a key position from a value one, asks this to see the same alternates TypeScript does.

pub fn bad(&self, why: impl Into<String>) -> Token

Construct a bad token at the current cursor.

pub fn bad_span( &self, why: impl Into<String>, start: usize, end: usize, ) -> Token

Construct a bad token whose displayed source is a scalar-indexed span. As in TypeScript, the diagnostic point remains the live cursor.

pub fn next_token(&mut self) -> Result<Token, TabnasError>

Fetches the next non-IGNORE token (skipping spaces, lines, comments).

pub fn next_raw_token(&mut self) -> Result<Token, TabnasError>

Fetch the next token without discarding whitespace, line, or comment tokens.

pub fn next_raw_for_rule( &mut self, rule: &mut Rule, context: &mut Context, ) -> Result<Token, Box<TabnasError>>

Fetch one token for an imperative parser callback, preserving ignored space/line/comment tokens just like TypeScript’s public lex.next. Replayed tokens produced by Context::rewind are served first.

pub fn next_for_rule( &mut self, rule: &mut Rule, context: &mut Context, ) -> Result<Token, Box<TabnasError>>

Fetch the next non-ignored token for an imperative parser callback.

pub fn relex_for_rule( &mut self, from: &Token, wanted: &[Tin], rule: &mut Rule, context: &mut Context, ) -> Option<(Token, RelexCheckpoint)>

Public negotiated-relex entry point for native parser callbacks. A successful recut commits the lexer cursor and returns an opaque undo checkpoint; a failed recut restores all lexer state before returning.

pub fn unrelex(&mut self, checkpoint: RelexCheckpoint, context: &mut Context)

Undo a committed Lexer::relex_for_rule operation, including the pending tokens hidden while the replacement cut was negotiated.