Function tokenize 

pub fn tokenize<'a>(
    input: OffsetStrIter<'a>,
    comment_map: Option<&mut CommentMap>,
) -> Result<Vec<Token>, BuildError>

Consumes an input OffsetStrIter and returns either a Vec or a error::Error. If a comment_map is passed in then it will store the comments indexed by their line number.