Module rustlr::lexer_interface

Rustlr allows the use of any lexical analyzer (tokenizer) that satisfies the Lexer trait. Only a simple charlexer tokenizer that separates non-whitespaces characters is provided as an example.

Structs

Lextoken

This structure is expected to be returned by the lexical analyzer (Lexer objects). Furthermore, the .sym field of a Lextoken must match the name of a terminal symbol specified in the grammar that defines the language. AT is the type of the value attached to the token, which is usually some enum that distinguishes between numbers, keywords, alphanumeric symbols and other symbols. See the tutorial and examples on how to define the right kind of AT type.

charlexer

This is a sample Lexer implementation designed to return every character in a string as a separate token, and is used in small grammars for testing and illustration purposes. It is assumed that the characters read are defined as terminal symbols in the grammar.

Traits

Lexer

This trait defines the interace that any lexical analyzer must be adopted to. The default implementations for linenum, column and current_line should be replaced. They’re provided only for compatibility.