Function tokenize 

pub fn tokenize(input: &str) -> Vec<Token>

Tokenizes input into a Vec<Token>, ending with a single Token::Eof. Convenience wrapper around Tokenizer for callers that don’t need streaming/lazy tokenization.