Module lexer 

A total, lossless lexer for BibTeX/BibLaTeX surface syntax.

Every byte of the input ends up in exactly one token, so concatenating all token texts reproduces the input verbatim — the losslessness invariant.

The lexer is context-free: brace/quote structure is the parser’s job (it tracks brace depth to build BRACE_GROUP / QUOTED value nodes), exactly as the LaTeX lexer leaves {/} grouping to its grammar. BibTeX needs no verbatim or catcode modes, so this lexer is simpler than the LaTeX one.

The specials @ { } ( ) , = # " % each lex as a single-character token; runs of whitespace, line breaks, and “word” characters (anything else) coalesce. A word run made up solely of ASCII digits is classified SyntaxKind::NUMBER, else SyntaxKind::WORD — so a later formatter / linter can tell an unquoted number from a macro name.

% is a bare SyntaxKind::PERCENT token here, not a comment: whether it opens a comment depends on brace/quote context ({50% off} is literal text), and that context is the grammar’s to know. The grammar wraps the run from a % to the end of its line in a SyntaxKind::COMMENT node only where BibTeX allows one.

Structs

Token

A single lexed token: its kind plus the exact source slice it covers.

Functions

lex

Lex input into a flat, lossless token stream.