Module tokenize 

The tokenizer, matched to DuckDB’s by behaviour.

This is the one part of the front end with no declarative artifact behind it. The grammar is vendored and generated from, per spec/20-the-grammar.md, and it says nothing about string literals, dollar quoting, numeric literal forms, comments or the operator rules. All of that is 613 lines of hand written C++ upstream, in src/parser/peg/tokenizer/base_tokenizer.cpp and parser_tokenizer.cpp, and this is a port of it rather than an interpretation.

Which is worth saying plainly: everything here is a fact about DuckDB and not a design decision of ours. Where the behaviour looks wrong, it is still the behaviour, because a query that returns a different answer is worse than a query that returns a surprising one. Section 20.7 enumerates the ten that bite. The reason a differential fuzzer against a real DuckDB is scheduled from week one rather than at M5 is this file.

One deliberate divergence, and it is representational. Upstream types a quoted identifier and a bare one both as IDENTIFIER and recovers the difference from the first byte where it matters. We give them separate kinds. Nothing downstream may treat them differently in a place upstream does not.

Nothing is decoded. A string keeps its quotes and its escapes, a number keeps its underscores, an identifier keeps its case. spec/20-the-grammar.md section 7 for why case in particular: DuckDB is case insensitive and case preserving, including for quoted identifiers, which is not what PostgreSQL does and not what folding here would give.

Functions

classes

The classes word belongs to, or zero.

lookup

The index of word in the generated keyword table, or NOT_A_KEYWORD.

tokenize

Split query into tokens, ending with exactly one Kind::EndOfInput.