Function tokenize 

pub fn tokenize(input: &str) -> Lexed<'_>

Tokenize Snowflake SQL into a lossless token stream.

The concatenation of tokens[i].text always equals input.