Struct BaseLexer 

pub struct BaseLexer<I> { /* private fields */ }

Implementations

impl<I> BaseLexer<I>

where I: CharStream,

pub fn new(input: I, data: RecognizerData) -> Self

pub fn reset(&mut self)

Resets runtime-owned lexer state so this instance can consume its input again from the beginning.

Learned DFA tables and configuration such as forced interpretation are retained. Token-production state, diagnostics, pending tokens, modes, and source position are cleared.

pub fn set_input_stream(&mut self, input: I)

Replaces the character stream and fully resets lexer state for reuse.

Learned DFA tables and configuration such as forced interpretation are retained. The new stream is always rewound to its beginning.

pub fn with_shared_dfa(self, atn: &'static LexerAtn) -> Self

Switches this lexer to the thread-shared learned DFA for atn.

Generated lexers create a fresh instance per parse; without sharing, every instance relearns the same DFA through ATN simulation. The shared cache is keyed by the generated lexer’s &'static LexerAtn identity and holds only input-independent data, so it stays valid across inputs. The showDFA edge trace lives in the cache too, so it reports the accumulated DFA — the same view the reference runtimes print from their static shared DFA.

pub fn clear_dfa(&self)

Clears the learned lexer DFA shared by recognizers for this grammar.

Ahead-of-time compiled DFA tables are immutable generated data and are unaffected. Any path that falls back to ATN interpretation relearns its dynamic DFA from an empty cache after this call.

pub const fn input(&self) -> &I

pub const fn input_mut(&mut self) -> &mut I

pub fn begin_token(&mut self)

Captures the input index and source position for the token currently being matched.

pub const fn token_start(&self) -> usize

Returns the absolute character index where the current token began.

pub const fn token_start_line(&self) -> usize

Returns the source line captured at the start of the current token.

pub const fn token_start_column(&self) -> usize

Returns the source column captured at the start of the current token.

pub const fn token_type(&self) -> i32

Returns the pending type of the token being matched.

pub const fn set_type(&mut self, token_type: i32)

Overrides the pending type of the token being matched.

pub const fn channel(&self) -> i32

Returns the pending channel of the token being matched.

pub const fn set_channel(&mut self, channel: i32)

Overrides the pending channel of the token being matched.

pub const fn skip(&mut self)

Marks the current match as skipped.

pub const fn more(&mut self)

Extends the current token with another lexer-rule match.

pub fn la(&mut self, offset: isize) -> i32

Reads a character at a one-based lookahead/lookbehind offset from the committed input cursor without moving it.

pub fn consume_char(&mut self)

Consumes one character from the input stream and updates lexer line and column counters.

The input stream is indexed by Unicode scalar values. Newline handling follows ANTLR’s default convention of incrementing the line and resetting the column after \n.

pub fn reset_accept_position(&mut self, index: usize)

Rewinds or advances the input cursor to a token accept boundary.

Some generated lexers intentionally accept a longer path to disambiguate a token, then emit only the prefix and leave the suffix for the next token. Recomputing line/column from token_start keeps the visible lexer position consistent after moving the cursor backwards.

pub fn set_token_start(&mut self, index: usize) -> bool

Moves the current token start forward within the consumed input span.

Source line and column are advanced with the start, so a subsequently emitted suffix token carries the same coordinates it would have had if lexed independently.

pub fn emit( &self, sink: &mut TokenSink<'_>, token_type: i32, channel: i32, text: Option<String>, ) -> Result<TokenId, TokenStoreError>

Builds a token spanning from the current token start to the character before the input cursor.

When generated or interpreted lexer code does not supply explicit text, the base lexer captures the matched source interval so downstream token streams and parse trees can render token text without retaining a source pair object.

pub fn emit_with_stop( &self, sink: &mut TokenSink<'_>, token_type: i32, channel: i32, stop: usize, text: Option<String>, ) -> Result<TokenId, TokenStoreError>

Builds a token with an explicit stop index.

EOF-matching lexer rules do not consume a Unicode scalar value, so their stop index can be one before the current input index. The caller passes usize::MAX to represent ANTLR’s -1 stop index at empty input.

pub fn enqueue_token( &mut self, token_type: i32, channel: i32, stop: usize, text: Option<String>, )

Queues an additional token to be returned before the current match’s automatic token.

The token spans the current token start through stop (inclusive). text = None keeps the token source-backed when the input supports it.

pub fn token_text(&self) -> String

Returns the current token text from the token start through the input cursor.

pub fn token_text_until(&self, stop_exclusive: usize) -> String

Returns the current token text from the token start through stop_exclusive.

Lexer custom actions can occur before the accepted token is complete. The action event records the position where the transition fired, and generated action code uses this helper to render ANTLR’s Text() template at that exact point.

pub fn column_at(&self, position: usize) -> usize

Computes the zero-based source column at an absolute input position reached during prediction of the current token.

pub fn eof_token( &self, sink: &mut TokenSink<'_>, ) -> Result<TokenId, TokenStoreError>

Builds the synthetic EOF token at the current input cursor.

impl<I> BaseLexer<I>

where I: CharStream,

pub const fn line(&self) -> usize

pub const fn column(&self) -> usize

pub fn source_name(&self) -> &str

pub fn source_text(&self) -> Option<Rc<str>>

pub const fn hit_eof(&self) -> bool

pub const fn set_hit_eof(&mut self, hit_eof: bool)

pub const fn set_force_interpreted(&mut self, force_interpreted: bool)

Routes every token through ATN interpretation even when the generated lexer carries an ahead-of-time compiled DFA.

Interpretation is what learns the replayable DFA that Self::lexer_dfa_string reports, so harnesses asserting on the observed-DFA trace (ANTLR’s showDFA descriptors) enable this before lexing.

pub const fn force_interpreted(&self) -> bool

Whether compiled-DFA entry points must fall back to interpretation.

pub fn record_error( &self, line: usize, column: usize, message: impl Into<String>, )

Buffers a lexer diagnostic until the token stream consumer is ready to emit errors in parser-compatible order.

pub fn record_semantic_error( &self, action: bool, rule_index: usize, coordinate_index: usize, )

Records one fail-loud semantic-hook miss per coordinate and token start.

pub fn drain_errors(&mut self) -> Vec<TokenSourceError>

Returns and clears lexer diagnostics produced while fetching tokens.

pub fn record_lexer_dfa_edge(&self, from: usize, symbol: i32, to: usize)

Records a visible lexer DFA edge unless it was already observed.

pub fn lexer_dfa_string(&self) -> String

Serializes the observed default-mode lexer DFA in ANTLR’s text shape.

Trait Implementations

impl<I: Clone> Clone for BaseLexer<I>

fn clone(&self) -> BaseLexer<I>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<I: Debug> Debug for BaseLexer<I>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<I> Lexer for BaseLexer<I>

where I: CharStream,

fn mode(&self) -> i32

fn set_mode(&mut self, mode: i32)

fn push_mode(&mut self, mode: i32)

fn pop_mode(&mut self) -> Option<i32>

impl<I> Recognizer for BaseLexer<I>

where I: CharStream,

fn data(&self) -> &RecognizerData

fn data_mut(&mut self) -> &mut RecognizerData

fn grammar_file_name(&self) -> &str

fn rule_names(&self) -> &[String]

fn channel_names(&self) -> &[String]

fn mode_names(&self) -> &[String]

fn vocabulary(&self) -> &Vocabulary

fn state(&self) -> isize

fn set_state(&mut self, state: isize)

fn sempred(&mut self, _rule_index: usize, _pred_index: usize) -> bool

fn action(&mut self, _rule_index: usize, _action_index: usize)