Function commands::tokenizer::tokenize

pub fn tokenize(text: &str) -> Result<Vec<Token>, TokenizerError>

Tokenize a body of text.