Function ftml::tokenizer::tokenize

pub fn tokenize<'t>(log: &Logger, text: &'t str) -> Tokenization<'t>

Take an input string and produce a list of tokens for consumption by the parser.