Struct rustlr::lexer_interface::charscanner

pub struct charscanner<'t> {
    pub modify: fn(_: &'t str) -> &'t str,
    /* private fields */
}

This is a sample Lexer implementation designed to return every character in a string as a separate token, and is used in small grammars for testing and illustration purposes. It is assumed that the characters read are defined as terminal symbols in the grammar. This replaces charlexer using Tokenizer and RawToken.

Fields

modify: fn(_: &'t str) -> &'t str

function to modify char returned by nextsym, can be changed. charscanner::new sets this function initially to |x|{x.to_str()}. For example, some characters such as ‘{’ and ‘}’ cannot be used as terminal symbols of a grammar and must be translated into something like “LBRACE” and “RBRACE”

Implementations

impl<'t> charscanner<'t>

pub fn new(input: &'t str, kws: bool) -> charscanner<'t>

creates a charscanner with the option of keeping whitespace chars if kws=true

Trait Implementations

impl<'t, AT: Default> Tokenizer<'t, AT> for charscanner<'t>

The source code of this implementation of the Tokenizer trait also serves as an illustration of how the trait should be implemented.

fn linenum(&self) -> usize

returns current line number starting from 1

fn column(&self) -> usize

returns the index of the current char, starting from 1

fn current_line(&self) -> &str

returns slice of underlying data using std::str::Chars::as_str

fn nextsym(&mut self) -> Option<TerminalToken<'t, AT>>

retrieves the next TerminalToken, or None at end-of-stream.

fn position(&self) -> usize

returns the absolute character position of the tokenizer. The default implementation returns 0;

fn get_line(&self, i: usize) -> Option<&str>

Retrieves the ith line of the raw input, if line index i is valid. This function should be called after the tokenizer has completed its task of scanning and tokenizing the entire input, when generating diagnostic messages when evaluating the AST post-parsing. The default implementation returns None.

fn get_slice(&self, start: usize, end: usize) -> &str

Retrieves the source string slice at the indicated indices; returns the empty string if indices are invalid. The default implementation returns the empty string.

fn source(&self) -> &str

retrieves the source (such as filename or URL) of the tokenizer. The default implementation returns the empty string.

fn transform_wildcard(&self, t: TerminalToken<'t, AT>) -> TerminalToken<'t, AT>

For internal use only unless not using StrTokenizer. This is a call-back function from the parser and can only be implemented when the grammar and token types are known. It transforms a token to a token representing the wildcard “_”, with semantic value indicating its position in the text. The default implementation returns the same TerminalToken. This function is automatically overridden by the generated lexer when using the -genlex option.

fn next_tt(&mut self) -> TerminalToken<'t, AT>

returns next TerminalToken. This provided function calls nextsym but will return a TerminalToken with sym=“EOF” at end of stream, with value=AT::default(). The is the only provided function that should not be re-implemented.