Module lrpar::lex

Structs

LexError

Lexeme

A Lexeme represents a segment of the user's input that conforms to a known type. All lexemes have a starting position in the user's input: lexemes that result from error recovery, however, do not have a length (or, therefore, an end). This allows us to differentiate between lexemes that are always of zero length (which are required in some grammars) from lexemes that result from error recovery (where an error recovery algorithm can know the type that a lexeme should have been, but can't know what its contents should have been).

Traits

Lexer

Roughly speaking, Lexer is an iterator which collectively produces Lexemes, as well as collecting the newlines encountered so that it can later optionally answer queries of the form "what's the line and column number of lexeme L".