Struct LexicalAnalyzer

pub struct LexicalAnalyzer<LexemeType> { /* private fields */ }

A lexical analyzer.

A lexical analyzer is a computation unit (an automaton), that is capable of reading a stream of characters, and separating it into Lexemes: atomic sequences units of meaningful text, tokens. See crate::lex for more detail.

Implementations

impl<LexemeType> LexicalAnalyzer<LexemeType>

where LexemeType: Hash + Eq + Clone,

pub fn new( lexeme_descriptors: Vec<LexemeDescriptor<LexemeType>>, ) -> LexicalAnalyzer<LexemeType>

Builds a new LexicalAnalyzer.

The different LexemeTypes that the analyzer will be capable of recognizing are described by lexeme_descriptors.

pub fn analyze<'a>( &'a self, reader: &'a mut impl Reader<u8>, ) -> impl Iterator<Item = Lexeme<LexemeType>> + 'a

Parses a stream of input text specified by a reader, and yields the lexemes it is consists of.

Conflict Resolution

• Longer lexemes are prioritized.
• Among the matching LexemeTypes for a lexeme of a given length, priority is given to the LexemeType whose LexemeDescriptor was listed first during the LexicalAnalyzer’s construction (see the lexeme_descriptors argument of LexicalAnalyzer::new).

Panics

If no known LexemeType could be matched against a prefix of the remaining input.