Module ucglib::tokenizer [−][src]
The tokenization stage of the ucg compiler.
Structs
| TokenIter |
TokenIter wraps a slice of Tokens and implements the various necessary nom traits to use it as an input to nom parsers. |
Functions
| pos |
pos gets the current position from a TokenIter input without consuming it. |
| token_clone | |
| tokenize |
Consumes an input Span and returns either a Vec |
Type Definitions
| Span |