Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(input: &str) -> impl Iterator<Item = Token<'_>> + '_
Expand description

Tokenize the given input