Trait TokenSource 

pub trait TokenSource<'input, 'arena, TF>
where
    TF: TokenFactory<'input, 'arena> + 'arena,
    'input: 'arena,
{
    // Required methods
    fn next_token(&mut self) -> &'arena mut TF::Tok;
    fn get_input_stream(&mut self) -> Option<&mut dyn IntStream>;
    fn get_source_name(&self) -> String;
    fn get_token_factory(&self) -> &TF;
    fn get_dfa_string(&self) -> String;

    // Provided methods
    fn get_line(&self) -> u32 { ... }
    fn get_char_position_in_line(&self) -> i32 { ... }
}

Produces tokens to be used by parser (aka Lexer).

TokenStream implementations are responsible for buffering tokens for parser lookahead

Required Methods

fn next_token(&mut self) -> &'arena mut TF::Tok

Return a {@link Token} object from your input stream (usually a {@link CharStream}). Do not fail/return upon lexing error; keep chewing on the characters until you get a good one; errors are not passed through to the parser.

fn get_input_stream(&mut self) -> Option<&mut dyn IntStream>

Returns underlying input stream

fn get_source_name(&self) -> String

Returns string identifier of underlying input e.g. file name

fn get_token_factory(&self) -> &TF

Gets the TokenFactory this token source is currently using for creating Token objects from the input.

Required by Parser for creating missing tokens.

fn get_dfa_string(&self) -> String

Provided Methods

fn get_line(&self) -> u32

Get the line number for the current position in the input stream. The first line in the input is line 1.

Returns the line number for the current position in the input stream, or 0 if the current token source does not track line numbers.

fn get_char_position_in_line(&self) -> i32

Get the index into the current line for the current position in the input stream. The first character on a line has position 0.

Returns the line number for the current position in the input stream, or -1 if the current token source does not track character positions.

Implementations on Foreign Types

impl<'input, 'arena, T, TF> TokenSource<'input, 'arena, TF> for &mut T

where T: TokenSource<'input, 'arena, TF>, TF: TokenFactory<'input, 'arena> + 'arena, 'input: 'arena,

fn next_token(&mut self) -> &'arena mut TF::Tok

fn get_line(&self) -> u32

fn get_char_position_in_line(&self) -> i32

fn get_input_stream(&mut self) -> Option<&mut dyn IntStream>

fn get_source_name(&self) -> String

fn get_token_factory(&self) -> &TF

fn get_dfa_string(&self) -> String

Implementors

impl<'input, 'arena, L, Input, TF> TokenSource<'input, 'arena, TF> for BaseLexer<'input, 'arena, L, Input, TF>

where L: LexerRecog<'input, 'arena, TF, Self> + 'static, Input: CharStream<'input>, TF: TokenFactory<'input, 'arena> + 'arena, 'input: 'arena,