Module ucglib::tokenizer

The tokenization stage of the ucg compiler.

Functions

pos	pos gets the current position from a TokenIter input without consuming it.
token_clone	Clones a token.
tokenize	Consumes an input OffsetStrIter and returns either a Vec or a error::Error.