Struct BaseLexer 

pub struct BaseLexer<I, F = CommonTokenFactory> { /* private fields */ }

Implementations

impl<I> BaseLexer<I>

where I: CharStream,

pub fn new(input: I, data: RecognizerData) -> Self

Creates a lexer base using CommonTokenFactory.

impl<I, F> BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

pub fn with_factory(input: I, data: RecognizerData, factory: F) -> Self

Creates a lexer base with a custom token factory.

pub fn with_shared_dfa(self, atn: &'static Atn) -> Self

Switches this lexer to the thread-shared learned DFA for atn.

Generated lexers create a fresh instance per parse; without sharing, every instance relearns the same DFA through ATN simulation. The shared cache is keyed by the generated lexer’s &'static Atn identity and holds only input-independent data, so it stays valid across inputs. The showDFA edge trace lives in the cache too, so it reports the accumulated DFA — the same view the reference runtimes print from their static shared DFA.

pub const fn input(&self) -> &I

pub const fn input_mut(&mut self) -> &mut I

pub fn begin_token(&mut self)

Captures the input index and source position for the token currently being matched.

pub const fn token_start(&self) -> usize

Returns the absolute character index where the current token began.

pub const fn token_start_line(&self) -> usize

Returns the source line captured at the start of the current token.

pub const fn token_start_column(&self) -> usize

Returns the source column captured at the start of the current token.

pub fn consume_char(&mut self)

Consumes one character from the input stream and updates lexer line and column counters.

The input stream is indexed by Unicode scalar values. Newline handling follows ANTLR’s default convention of incrementing the line and resetting the column after \n.

pub fn reset_accept_position(&mut self, index: usize)

Rewinds or advances the input cursor to a token accept boundary.

Some generated lexers intentionally accept a longer path to disambiguate a token, then emit only the prefix and leave the suffix for the next token. Recomputing line/column from token_start keeps the visible lexer position consistent after moving the cursor backwards.

pub fn emit( &self, token_type: i32, channel: i32, text: Option<String>, ) -> CommonToken

Builds a token spanning from the current token start to the character before the input cursor.

When generated or interpreted lexer code does not supply explicit text, the base lexer captures the matched source interval so downstream token streams and parse trees can render token text without retaining a source pair object.

pub fn emit_with_stop( &self, token_type: i32, channel: i32, stop: usize, text: Option<String>, ) -> CommonToken

Builds a token with an explicit stop index.

EOF-matching lexer rules do not consume a Unicode scalar value, so their stop index can be one before the current input index. The caller passes usize::MAX to represent ANTLR’s -1 stop index at empty input.

pub fn token_text(&self) -> String

Returns the current token text from the token start through the input cursor.

pub fn token_text_until(&self, stop_exclusive: usize) -> String

Returns the current token text from the token start through stop_exclusive.

Lexer custom actions can occur before the accepted token is complete. The action event records the position where the transition fired, and generated action code uses this helper to render ANTLR’s Text() template at that exact point.

pub fn column_at(&self, position: usize) -> usize

Computes the zero-based source column at an absolute input position reached during prediction of the current token.

pub fn eof_token(&self) -> CommonToken

Builds the synthetic EOF token at the current input cursor.

impl<I, F> BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

pub const fn line(&self) -> usize

pub const fn column(&self) -> usize

pub fn source_name(&self) -> &str

pub const fn hit_eof(&self) -> bool

pub const fn set_hit_eof(&mut self, hit_eof: bool)

pub const fn set_force_interpreted(&mut self, force_interpreted: bool)

Routes every token through ATN interpretation even when the generated lexer carries an ahead-of-time compiled DFA.

Interpretation is what learns the replayable DFA that Self::lexer_dfa_string reports, so harnesses asserting on the observed-DFA trace (ANTLR’s showDFA descriptors) enable this before lexing.

pub const fn force_interpreted(&self) -> bool

Whether compiled-DFA entry points must fall back to interpretation.

pub fn record_error( &self, line: usize, column: usize, message: impl Into<String>, )

Buffers a lexer diagnostic until the token stream consumer is ready to emit errors in parser-compatible order.

pub fn record_semantic_error( &self, action: bool, rule_index: usize, coordinate_index: usize, )

Records one fail-loud semantic-hook miss per coordinate and token start.

pub fn drain_errors(&mut self) -> Vec<TokenSourceError>

Returns and clears lexer diagnostics produced while fetching tokens.

pub fn record_lexer_dfa_edge(&self, from: usize, symbol: i32, to: usize)

Records a visible lexer DFA edge unless it was already observed.

pub fn lexer_dfa_string(&self) -> String

Serializes the observed default-mode lexer DFA in ANTLR’s text shape.

Trait Implementations

impl<I: Clone, F: Clone> Clone for BaseLexer<I, F>

fn clone(&self) -> BaseLexer<I, F>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<I: Debug, F: Debug> Debug for BaseLexer<I, F>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<I, F> Lexer for BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

fn mode(&self) -> i32

fn set_mode(&mut self, mode: i32)

fn push_mode(&mut self, mode: i32)

fn pop_mode(&mut self) -> Option<i32>

impl<I, F> Recognizer for BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

fn data(&self) -> &RecognizerData

fn data_mut(&mut self) -> &mut RecognizerData

fn grammar_file_name(&self) -> &str

fn rule_names(&self) -> &[String]

fn channel_names(&self) -> &[String]

fn mode_names(&self) -> &[String]

fn vocabulary(&self) -> &Vocabulary

fn state(&self) -> isize

fn set_state(&mut self, state: isize)

fn sempred(&mut self, _rule_index: usize, _pred_index: usize) -> bool

fn action(&mut self, _rule_index: usize, _action_index: usize)