Struct BaseLexer 

pub struct BaseLexer<I, F = CommonTokenFactory> { /* private fields */ }

Implementations

impl<I> BaseLexer<I>

where I: CharStream,

pub const fn new(input: I, data: RecognizerData) -> Self

Creates a lexer base using CommonTokenFactory.

impl<I, F> BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

pub const fn with_factory(input: I, data: RecognizerData, factory: F) -> Self

Creates a lexer base with a custom token factory.

pub const fn input(&self) -> &I

pub const fn input_mut(&mut self) -> &mut I

pub fn begin_token(&mut self)

Captures the input index and source position for the token currently being matched.

pub const fn token_start(&self) -> usize

Returns the absolute character index where the current token began.

pub const fn token_start_line(&self) -> usize

Returns the source line captured at the start of the current token.

pub const fn token_start_column(&self) -> usize

Returns the source column captured at the start of the current token.

pub fn consume_char(&mut self)

Consumes one character from the input stream and updates lexer line and column counters.

The input stream is indexed by Unicode scalar values. Newline handling follows ANTLR’s default convention of incrementing the line and resetting the column after \n.

pub fn reset_accept_position(&mut self, index: usize)

Rewinds or advances the input cursor to a token accept boundary.

Some generated lexers intentionally accept a longer path to disambiguate a token, then emit only the prefix and leave the suffix for the next token. Recomputing line/column from token_start keeps the visible lexer position consistent after moving the cursor backwards.

pub fn emit( &self, token_type: i32, channel: i32, text: Option<String>, ) -> CommonToken

Builds a token spanning from the current token start to the character before the input cursor.

When generated or interpreted lexer code does not supply explicit text, the base lexer captures the matched source interval so downstream token streams and parse trees can render token text without retaining a source pair object.

pub fn emit_with_stop( &self, token_type: i32, channel: i32, stop: usize, text: Option<String>, ) -> CommonToken

Builds a token with an explicit stop index.

EOF-matching lexer rules do not consume a Unicode scalar value, so their stop index can be one before the current input index. The caller passes usize::MAX to represent ANTLR’s -1 stop index at empty input.

pub fn token_text(&self) -> String

Returns the current token text from the token start through the input cursor.

pub fn token_text_until(&self, stop_exclusive: usize) -> String

Returns the current token text from the token start through stop_exclusive.

Lexer custom actions can occur before the accepted token is complete. The action event records the position where the transition fired, and generated action code uses this helper to render ANTLR’s Text() template at that exact point.

pub fn column_at(&self, position: usize) -> usize

Computes the zero-based source column at an absolute input position reached during prediction of the current token.

pub fn eof_token(&self) -> CommonToken

Builds the synthetic EOF token at the current input cursor.

impl<I, F> BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

pub const fn line(&self) -> usize

pub const fn column(&self) -> usize

pub fn source_name(&self) -> &str

pub const fn hit_eof(&self) -> bool

pub const fn set_hit_eof(&mut self, hit_eof: bool)

pub fn record_error( &mut self, line: usize, column: usize, message: impl Into<String>, )

Buffers a lexer diagnostic until the token stream consumer is ready to emit errors in parser-compatible order.

pub fn drain_errors(&mut self) -> Vec<TokenSourceError>

Returns and clears lexer diagnostics produced while fetching tokens.

pub fn record_lexer_dfa_edge(&mut self, from: usize, symbol: i32, to: usize)

Records a visible lexer DFA edge unless it was already observed.

pub fn lexer_dfa_string(&self) -> String

Serializes the observed default-mode lexer DFA in ANTLR’s text shape.

Trait Implementations

impl<I: Clone, F: Clone> Clone for BaseLexer<I, F>

fn clone(&self) -> BaseLexer<I, F>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<I: Debug, F: Debug> Debug for BaseLexer<I, F>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<I, F> Lexer for BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

fn mode(&self) -> i32

fn set_mode(&mut self, mode: i32)

fn push_mode(&mut self, mode: i32)

fn pop_mode(&mut self) -> Option<i32>

impl<I, F> Recognizer for BaseLexer<I, F>

where I: CharStream, F: TokenFactory,

fn data(&self) -> &RecognizerData

fn data_mut(&mut self) -> &mut RecognizerData

fn grammar_file_name(&self) -> &str

fn rule_names(&self) -> &[String]

fn channel_names(&self) -> &[String]

fn mode_names(&self) -> &[String]

fn vocabulary(&self) -> &Vocabulary

fn state(&self) -> isize

fn set_state(&mut self, state: isize)

fn sempred(&mut self, _rule_index: usize, _pred_index: usize) -> bool

fn action(&mut self, _rule_index: usize, _action_index: usize)