Module lexer 

Lexer for the query language.

Produces span-based tokens without storing text - text is sliced from source only when needed.

Error handling

The lexer coalesces consecutive error characters into single Garbage tokens rather than producing one error per character. This keeps the token stream manageable for malformed input.

Structs

Token

Zero-copy token: kind + span, text retrieved via token_text when needed.

Functions

lex

Tokenizes source into a vector of span-based tokens.

token_text

Retrieves the text slice for a token. O(1) slice into source.