Function ucglib::tokenizer::tokenize

pub fn tokenize<'a>(input: OffsetStrIter<'a>) -> Result<Vec<Token>, String>

Consumes an input OffsetStrIter and returns either a Vec or a error::Error.