Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(kind: TokenizerKind, text: &str) -> Vec<Token>