[−][src]Module ucglib::tokenizer
The tokenization stage of the ucg compiler.
Functions
pos | pos gets the current position from a TokenIter input without consuming it. |
token_clone | Clones a token. |
tokenize | Consumes an input OffsetStrIter and returns either a Vec |
Type Definitions
CommentGroup | |
CommentMap |