pub fn tokenize(input: &str) -> Result<Vec<Token<'_>>, Err<Error<Text<'_>>>>Expand description
Tokenize an EventQL query string.
This function performs lexical analysis on the input string, converting it into a sequence of tokens. Each token includes position information (line and column numbers) for error reporting.
ยงRecognized Tokens
- Identifiers: Alphanumeric names starting with a letter (e.g.,
events,e) - Keywords: Case-insensitive SQL-like keywords detected by the parser
- Numbers: Floating-point literals (e.g.,
42,3.14) - Strings: Double-quoted string literals (e.g.,
"hello") - Operators: Arithmetic (
+,-,*,/), comparison (==,!=,<,<=,>,>=), logical (AND,OR,XOR,NOT) - Symbols: Structural characters (
(,),[,],{,},.,,,:)