Expand description
Contains a set of custom tokens and a tokenizer
Structs§
- Token
Stream - A TokenStream is a string of tokens with methods to fill, iterate and parse the tokens.
Enums§
- Parse
Token Error - The two possible errors from the tokenizer is an invalid token or some I/O error
- Token