Function ucglib::tokenizer::tokenize

pub fn tokenize<'a>(
    input: OffsetStrIter<'a>, 
    comment_map: Option<&mut CommentMap>
) -> Result<Vec<Token>, BuildError>

Consumes an input OffsetStrIter and returns either a Vec or a error::Error. If a comment_map is passed in then it will store the comments indexed by their line number.