[−][src]Module ucglib::tokenizer

The tokenization stage of the ucg compiler.

Functions

pos

pos gets the current position from a TokenIter input without consuming it.

token_clone

Clones a token.

tokenize

Consumes an input OffsetStrIter and returns either a Vec or a error::Error. If a comment_map is passed in then it will store the comments indexed by their line number.

Type Definitions

CommentGroup
CommentMap