Module ucglib::tokenizer [−][src]
The tokenization stage of the ucg compiler.
Structs
TokenIter |
TokenIter wraps a slice of Tokens and implements the various necessary nom traits to use it as an input to nom parsers. |
Functions
pos |
pos gets the current position from a TokenIter input without consuming it. |
token_clone |
Clones a token. |
tokenize |
Consumes an input Span and returns either a Vec |
Type Definitions
Span |