Trait TokenSource 

pub trait TokenSource<'input> {
    type TF: TokenFactory<'input> + 'input;

    // Required methods
    fn next_token(&mut self) -> <Self::TF as TokenFactory<'input>>::Tok;
    fn get_input_stream(&mut self) -> Option<&mut dyn IntStream>;
    fn get_source_name(&self) -> String;
    fn get_token_factory(&self) -> &'input Self::TF;

    // Provided methods
    fn get_line(&self) -> isize { ... }
    fn get_char_position_in_line(&self) -> isize { ... }
}

Produces tokens to be used by parser. TokenStream implementations are responsible for buffering tokens for parser lookahead

Required Associated Types

type TF: TokenFactory<'input> + 'input

TokenFactory this token source produce tokens with

Required Methods

fn next_token(&mut self) -> <Self::TF as TokenFactory<'input>>::Tok

Return a {@link Token} object from your input stream (usually a {@link CharStream}). Do not fail/return upon lexing error; keep chewing on the characters until you get a good one; errors are not passed through to the parser.

fn get_input_stream(&mut self) -> Option<&mut dyn IntStream>

Returns underlying input stream

fn get_source_name(&self) -> String

Returns string identifier of underlying input e.g. file name

fn get_token_factory(&self) -> &'input Self::TF

Gets the TokenFactory this token source is currently using for creating Token objects from the input.

Required by Parser for creating missing tokens.

Provided Methods

fn get_line(&self) -> isize

Get the line number for the current position in the input stream. The first line in the input is line 1.

Returns the line number for the current position in the input stream, or 0 if the current token source does not track line numbers.

fn get_char_position_in_line(&self) -> isize

Get the index into the current line for the current position in the input stream. The first character on a line has position 0.

Returns the line number for the current position in the input stream, or -1 if the current token source does not track character positions.

Implementations on Foreign Types

impl<'input, T> TokenSource<'input> for &mut T

where T: TokenSource<'input>,

type TF = <T as TokenSource<'input>>::TF

fn next_token(&mut self) -> <Self::TF as TokenFactory<'input>>::Tok

fn get_line(&self) -> isize

fn get_char_position_in_line(&self) -> isize

fn get_input_stream(&mut self) -> Option<&mut dyn IntStream>

fn get_source_name(&self) -> String

fn get_token_factory(&self) -> &'input Self::TF

Implementors

impl<'input, T, Input, TF> TokenSource<'input> for BaseLexer<'input, T, Input, TF>

where T: LexerRecog<'input, Self> + 'static, Input: CharStream<TF::From>, TF: TokenFactory<'input>,

type TF = TF