Expand description
Turning spec text into tokens.
§Why a lexer instead of splitting on punctuation
The previous parser split statements on ; and lists on ,. That works
until a value contains one — and a description is prose, so it contains
semicolons routinely:
description = "Use when summarizing; especially long files.";Splitting on ; cuts that in half and reports a confusing error about the
description not being a quoted string. A path containing a comma broke the
capability list the same way. Both were real bugs, not hypotheticals, and
neither is fixable by being cleverer about splitting: a separator inside a
string is only distinguishable from a separator between values by tracking
whether you are inside a string, which is what a lexer is.
It also buys positions. “malformed spec” with no location is a poor error for a file someone is editing by hand; every token here carries a line and column so the parser can point at the problem.
Structs§
Enums§
Functions§
- lex
- Tokenize spec source.