Expand description
A total, lossless lexer for BibTeX/BibLaTeX surface syntax.
Every byte of the input ends up in exactly one token, so concatenating all token texts reproduces the input verbatim — the losslessness invariant.
The lexer is context-free: brace/quote structure is the parser’s job (it
tracks brace depth to build BRACE_GROUP / QUOTED value nodes), exactly as
the LaTeX lexer leaves {/} grouping to its grammar. BibTeX needs no
verbatim or catcode modes, so this lexer is simpler than the LaTeX one.
The specials @ { } ( ) , = # " each lex as a single-character token; runs of
whitespace, line breaks, and “word” characters (anything else) coalesce. A
word run made up solely of ASCII digits is classified SyntaxKind::NUMBER,
else SyntaxKind::WORD — so a later formatter / linter can tell an unquoted
number from a macro name.
Structs§
- Token
- A single lexed token: its kind plus the exact source slice it covers.
Functions§
- lex
- Lex
inputinto a flat, lossless token stream.