Trait rustemo::Lexer

pub trait Lexer<'i, C, S, TK>
where
    C: Context<'i, Self::Input, S, TK>,
    S: State,
{
    type Input: Input + ?Sized;

    // Required method
    fn next_tokens(
        &self,
        context: &mut C,
        input: &'i Self::Input,
        expected_tokens: Vec<(TK, bool)>
    ) -> Box<dyn Iterator<Item = Token<'i, Self::Input, TK>> + 'i>;
}

The trait implemented by all Rustemo lexers

Lexer is stateless and its job is to produce the next token given the current context.

Generic types

• C - parsing context
• S - parser state type.
• TK - token kind type. This is generated by the parser generator from the grammar. This is the type that describes the kinds of token lexer can produce.

Required Associated Types

type Input: Input + ?Sized

Required Methods

fn next_tokens( &self, context: &mut C, input: &'i Self::Input, expected_tokens: Vec<(TK, bool)> ) -> Box<dyn Iterator<Item = Token<'i, Self::Input, TK>> + 'i>

Given the current context, this method yield an iterator over possible tokens found at the current location where the order and kinds of token to look for, and its finish flags, are given by the expected_tokens parameter.

Context is mutable to support lexers that implement skipping of whitespaces.

Object Safety

This trait is not object safe.

Implementors

impl<'i, C, S, TK, TR, const TERMINAL_COUNT: usize> Lexer<'i, C, S, TK> for StringLexer<C, S, TK, TR, TERMINAL_COUNT>

where C: Context<'i, str, S, TK>, S: State + Into<usize>, TK: Debug + Into<usize> + Copy + 'i, TR: TokenRecognizer<'i>,

type Input = str