Module lexer 

A total, lossless lexer for LaTeX surface syntax.

Every byte of the input ends up in exactly one token, so concatenating all token texts reproduces the input verbatim — the losslessness invariant. The lexer is mostly context-free, with a small set of statically recognizable modes:

• \verb / \verb* inline verbatim: the delimited argument is consumed as a single SyntaxKind::VERB token (otherwise the delimiters glue into ordinary WORD runs and become un-splittable downstream).
• verbatim-like environments (verbatim, lstlisting, minted, …): the body between \begin{name} and \end{name} is one SyntaxKind::VERBATIM_BODY token, so %, $, \ inside are never (mis)lexed as comments / math. For argument-taking ones the \begin arguments are tokenized first (the built-in signature DB says where the raw body starts); see [lex_verbatim_environment].
• \makeatletter / \makeatother: toggles @ into a letter so that \foo@bar lexes as one control word.
• \ExplSyntaxOn / \ExplSyntaxOff (also opened by \ProvidesExplPackage / \ProvidesExplClass / \ProvidesExplFile): toggles _ and : into letters so expl3 names (\seq_new:N, \__module_internal:nn) lex as one control word. Composes with \makeatletter for the @@ module-prefix convention (\g_@@_frame_title_tl).
• \left / \right delimiters: the single delimiter that follows is isolated as its own token, so a word-character delimiter ((, ), |, /, ., <, >) does not glue into the following word run and become un-splittable downstream (the same problem \verb has). Control-symbol / control-word / bracket delimiters already lex as single tokens.

None of these resolve macro meaning; they are surface lexing concerns (in TeX, catcodes genuinely change in these regions).

Structs

LexConfig

The lexer’s per-parse mode. flavor fixes the initial catcode regime (a .sty/.cls starts under an implicit \makeatletter), while dtx is an orthogonal axis: when set, the lexer runs the bounded line-oriented docstrip mode for a .dtx file — line-leading % margins become DOC_MARGIN trivia, line-leading %<…> guards become GUARD trivia, and macrocode bodies lex as ordinary code (AGENTS.md decision #1). The two axes are independent because a .dtx’s catcode regime varies by layer (its documentation is Document-flavored, its macrocode Package-flavored), so dtx cannot be folded into a LatexFlavor variant.

ParseCtx

Per-parse context carrying the facts the parser can only learn by first scanning the file’s own definitions (crate::semantic::define) — the sanctioned second pass described in parser::core. Empty for the first pass; populated for the second when the document defines any. Both the lexer and the grammar read it, so the two can never disagree about what a name is.

Token

A single lexed token: its kind plus the exact source slice it covers.

Enums

ExplToggle

An expl3 catcode-mode toggle recognized purely by its control-word spelling. Shared by the lexer (which flips its expl_syntax flag) and the formatter’s region pre-pass (the badness-formatter crate recomputes in-region byte spans), so the two read the same fixed toggle set and can never drift.

LatexFlavor

The LaTeX file flavor, fixing the lexer’s initial catcode regime. A document (.tex) starts in the ordinary regime; a package or class (.sty/.cls) is loaded under an implicit \makeatletter, so @ is a letter from the first byte. A trailing explicit \makeatother still applies.

Functions

expl_toggle

Classify a control word’s text as an expl3 catcode-mode toggle, if any. Only meaningful on SyntaxKind::CONTROL_WORD text: a \ExplSyntaxOn inside a \verb/comment lexes as a VERB/COMMENT token and so never reaches here.

is_control_word_name

Could name (without its leading \) lex as a single control word in some catcode regime?

is_word_char

Ordinary text: anything that is not whitespace, a line break, or one of the characters the lexer treats specially.

lex

Lex input into a flat, lossless token stream, consulting only the built-in signature DB for verbatim commands/environments. The entry used by the first parse pass; lex_with adds user-defined verbatim commands. Uses the Document flavor (ordinary starting catcodes).

lex_with

Lex input like lex, additionally treating the user-defined verbatim commands in ctx as verbatim (their final argument captured as one VERB token). Used by the second parse pass once definition scanning has discovered catcode-othering commands. config fixes the initial catcode regime (a Package flavor starts with @ already a letter) and whether to run the .dtx docstrip mode.

Type Aliases

VerbCtx

The former name of ParseCtx, kept so the published crate’s API does not break. It carries environment aliases as well as verbatim facts now.