Expand description
The tokenization stage of the ucg compiler.
Functions§
- pos
- pos gets the current position from a TokenIter input without consuming it.
- token_
clone - Clones a token.
- tokenize
- Consumes an input OffsetStrIter and returns either a Vec
or a error::Error. If a comment_map is passed in then it will store the comments indexed by their line number.