Module lexer 

A total, lossless lexer for LaTeX surface syntax.

Every byte of the input ends up in exactly one token, so concatenating all token texts reproduces the input verbatim — the losslessness invariant. The lexer is mostly context-free, with three bounded, statically-recognizable modes sanctioned by AGENTS.md Core decision #1:

• \verb / \verb* inline verbatim: the delimited argument is consumed as a single SyntaxKind::VERB token (otherwise the delimiters glue into ordinary WORD runs and become un-splittable downstream).
• verbatim-like environments (verbatim, lstlisting, minted, …): the body between \begin{name} and \end{name} is one SyntaxKind::VERBATIM_BODY token, so %, $, \ inside are never (mis)lexed as comments / math. For argument-taking ones the \begin arguments are tokenized first (the built-in signature DB says where the raw body starts); see [lex_verbatim_environment].
• \makeatletter / \makeatother: toggles @ into a letter so that \foo@bar lexes as one control word.
• \ExplSyntaxOn / \ExplSyntaxOff (also opened by \ProvidesExplPackage / \ProvidesExplClass / \ProvidesExplFile): toggles _ and : into letters so expl3 names (\seq_new:N, \__module_internal:nn) lex as one control word. Composes with \makeatletter for the @@ module-prefix convention (\g_@@_frame_title_tl).
• \left / \right delimiters: the single delimiter that follows is isolated as its own token, so a word-character delimiter ((, ), |, /, ., <, >) does not glue into the following word run and become un-splittable downstream (the same problem \verb has). Control-symbol / control-word / bracket delimiters already lex as single tokens.

None of these resolve macro meaning; they are surface lexing concerns (in TeX, catcodes genuinely change in these regions).

Structs

LexConfig

The lexer’s per-parse mode. flavor fixes the initial catcode regime (a .sty/.cls starts under an implicit \makeatletter), while dtx is an orthogonal axis: when set, the lexer runs the bounded line-oriented docstrip mode for a .dtx file — line-leading % margins become DOC_MARGIN trivia, line-leading %<…> guards become GUARD trivia, and macrocode bodies lex as ordinary code (AGENTS.md decision #1). The two axes are independent because a .dtx’s catcode regime varies by layer (its documentation is Document-flavored, its macrocode Package-flavored), so dtx cannot be folded into a LatexFlavor variant.

Token

A single lexed token: its kind plus the exact source slice it covers.

VerbCtx

Per-parse lexer context carrying user-defined verbatim constructs — those a document declares with catcode manipulation (\@makeother\$, …), found by scanning definition bodies (crate::semantic::define). The lexer consults it (alongside the built-in DB) to capture a verbatim command’s final argument as one VERB token, and a verbatim environment’s body as one VERBATIM_BODY token. Empty for the first parse pass; populated for the second when the document defines any (see parser::core).

Enums

ExplToggle

An expl3 catcode-mode toggle recognized purely by its control-word spelling. Shared by the lexer (which flips its expl_syntax flag) and the formatter’s region pre-pass (the badness-formatter crate recomputes in-region byte spans), so the two read the same fixed toggle set and can never drift.

LatexFlavor

The LaTeX file flavor, fixing the lexer’s initial catcode regime. A document (.tex) starts in the ordinary regime; a package or class (.sty/.cls) is loaded under an implicit \makeatletter, so @ is a letter from the first byte (a static, extension-driven catcode fact — sanctioned exactly like the explicit \makeatletter mode, AGENTS.md decision #1). A trailing explicit \makeatother still applies.

Functions

expl_toggle

Classify a control word’s text as an expl3 catcode-mode toggle, if any. Only meaningful on SyntaxKind::CONTROL_WORD text: a \ExplSyntaxOn inside a \verb/comment lexes as a VERB/COMMENT token and so never reaches here.

is_word_char

Ordinary text: anything that is not whitespace, a line break, or one of the characters the lexer treats specially.

lex

Lex input into a flat, lossless token stream, consulting only the built-in signature DB for verbatim commands/environments. The entry used by the first parse pass; lex_with adds user-defined verbatim commands. Uses the Document flavor (ordinary starting catcodes).

lex_with

Lex input like lex, additionally treating the user-defined verbatim commands in ctx as verbatim (their final argument captured as one VERB token). Used by the second parse pass once definition scanning has discovered catcode-othering commands. config fixes the initial catcode regime (a Package flavor starts with @ already a letter) and whether to run the .dtx docstrip mode.