Module luthor::tokenizer
[−]
[src]
Structs
StateFunction | |
Tokenizer |
The Tokenizer type is used to produce and store tokens for the various language and format lexers. |
Functions
new |
Initializes a new tokenizer with the given data. |