Expand description
Core tokenizer for the language, defines all variants a token may have
Modules§
- keyword
- Keyword definitions and global randomizer
Structs§
- Token
- A token with respect to it’s location in the token stream
- Tokenize
Error - An error during tokenization
Enums§
- Token
Tag - All tags for a token
Traits§
- Tokenizable
- Any object that can be transformed into a token stream
Type Aliases§
- Result
- An result type for tokenizing