Trait rustlr::lexer_interface::Tokenizer

pub trait Tokenizer<'t, AT: Default> {
    fn nextsym(&mut self) -> Option<TerminalToken<'t, AT>>;

    fn linenum(&self) -> usize { ... }
    fn column(&self) -> usize { ... }
    fn position(&self) -> usize { ... }
    fn current_line(&self) -> &str { ... }
    fn get_line(&self, i: usize) -> Option<&str> { ... }
    fn source(&self) -> &str { ... }
    fn next_tt(&mut self) -> TerminalToken<'t, AT> { ... }
}

This is the trait that repesents an abstract lexical scanner for any grammar. Any tokenizer must be adopted to implement this trait. The default implementations of functions such as Tokenizer::linenum do not return correct values and should be replaced: they’re only given defaults for easy compatibility with prototypes that may not have their own implementations. This trait replaced LexToken used in earlier versions of Rustlr.

Required Methods

fn nextsym(&mut self) -> Option<TerminalToken<'t, AT>>

retrieves the next TerminalToken, or None at end-of-stream.

Provided Methods

fn linenum(&self) -> usize

returns the current line number. The default implementation returns 0.

fn column(&self) -> usize

returns the current column (character position) on the current line. The default implementation returns 0;

fn position(&self) -> usize

returns the absolute character position of the tokenizer. The default implementation returns 0;

fn current_line(&self) -> &str

returns the current line being tokenized. The default implementation returns the empty string.

fn get_line(&self, i: usize) -> Option<&str>

Retrieves the ith line of the raw input, if line index i is valid. This function should be called after the tokenizer has completed its task of scanning and tokenizing the entire input, when generating diagnostic messages when evaluating the AST post-parsing. The default implementation returns None.

fn source(&self) -> &str

retrieves the source (such as filename or URL) of the tokenizer. The default implementation returns the empty string.

fn next_tt(&mut self) -> TerminalToken<'t, AT>

returns next TerminalToken. This provided function calls nextsym but will return a TerminalToken with sym=“EOF” at end of stream, with value=AT::default(). The is the only provided function that should not be re-implemented.

Implementors

impl<'t, AT: Default> Tokenizer<'t, AT> for charscanner<'t>

The source code of this implementation of the Tokenizer trait also serves as an illustration of how the trait should be implemented.