Expand description
What the tokenizer produces and what the matcher consumes.
A token is twelve bytes and holds no string. The text stays in the query, the token holds a span into it, and nothing is decoded here: a string keeps its quotes and its escapes, a number keeps its underscores, an identifier keeps its case. Decoding is the transformer’s job, and leaving it there is what keeps the whole token vector in cache for a query of any sane size.
spec/20-the-grammar.md section 7 is the behaviour this has to match and why matching it is
the largest single compatibility risk in the front end.
Structs§
- Flags
- Facts about the gap before a token, which the grammar cannot see and two rules need.
- Token
- One token. Twelve bytes, no allocation, no owned text.
Enums§
- Kind
- What a token is.
Constants§
- NOT_
A_ KEYWORD - A word is not a keyword.