Trait rustlr::lexer_interface::Tokenizer

pub trait Tokenizer<'t, AT: Default> {
    fn nextsym(&mut self) -> Option<TerminalToken<'t, AT>>;

    fn linenum(&self) -> usize { ... }
    fn column(&self) -> usize { ... }
    fn position(&self) -> usize { ... }
    fn current_line(&self) -> &str { ... }
    fn source(&self) -> &str { ... }
    fn next_tt(&mut self) -> TerminalToken<'t, AT> { ... }
}

This trait is intended to replace Lexer, and won’t use owned strings

Required methods

fn nextsym(&mut self) -> Option<TerminalToken<'t, AT>>

retrieves the next TerminalToken, or None at end-of-stream.

Provided methods

fn linenum(&self) -> usize

returns the current line number. The default implementation returns 0.

fn column(&self) -> usize

returns the current column (character position) on the current line. The default implementation returns 0;

fn position(&self) -> usize

returns the absolute character position of the tokenizer. The default implementation returns 0;

fn current_line(&self) -> &str

returns the current line being tokenized. The default implementation returns the empty string.

fn source(&self) -> &str

retrieves the source (such as filename or URL) of the tokenizer. The default implementation returns the empty string.

fn next_tt(&mut self) -> TerminalToken<'t, AT>

returns next TerminalToken. This provided function calls nextsym but will return a TerminalToken with sym=“EOF” at end of stream, with value=AT::default(). The is the only provided function that should not be re-implemented.

Implementors

impl<'t, AT: Default> Tokenizer<'t, AT> for charscanner<'t>

The source code of this implementation of the Tokenizer trait also serves as an illustration of how the trait should be implemented.