tokenize

Function tokenize 

Source
pub fn tokenize(input: &str) -> Lexer<'_, TokenKind> 
Expand description

Returns an iterator over the token kinds produced by the tokenizer.