pub fn tokenize(query: &str) -> Result<Vec<Token>>Expand description
Split query into tokens, ending with exactly one Kind::EndOfInput.
Fails only where DuckDB’s parser tokenizer throws, which is four cases: an unterminated block
comment, an unterminated string literal, an unterminated quoted identifier, and the empty
quoted identifier "". An unterminated dollar quoted string is not one of them and comes back
as a token with Flags::UNTERMINATED set.