Module parser 

Tolerant LaTeX tokenizer — a from-scratch parser producing the same six node kinds as pylatexenc.latexwalker, tuned to match its observed behavior rather than general TeX semantics. Ground truth was established empirically against the installed pylatexenc (see git history / plan notes for the probe sessions), not by reading its source line-by-line — where the two might diverge, the tests/parser_parity.rs fixture-diff harness is the actual arbiter, per the plan’s Phase 1 scope (“no rules yet — just prove the substrate matches”).

Key empirically-verified rules this parser encodes:

1. Control-word whitespace eating. A control WORD (\ + one or more [A-Za-z@]) swallows trailing spaces/tabs and at most one newline (plus further spaces/tabs after it) into its own span — UNLESS eating that newline would immediately cross a blank line (two adjacent \ns), which is left untouched as a paragraph boundary. A control SYMBOL (\ + exactly one other character) does not eat anything.
2. Unknown macros take zero arguments. A macro/environment name absent from specs::macro_argspec (compiled from pylatexenc’s own default database — see tex/specs.rs) ends immediately after its (whitespace-eaten) name. Whatever follows — commonly a {...} group — is simply the next sibling node in the parent’s list, not an argument. This is why JSS’s own macros (\pkg, \proglang, \code, …) need the sibling-group fallback that tex::prose ports from _helpers.py::_macro_args_text.
3. Known macros/environments parse arguments per an argspec string ({ mandatory group, [ optional bracket group, * optional literal star), each preceded by a TeX-whitespace skip (rule 1’s skipping logic, reused) — except a [ slot that finds no [ does NOT consume the whitespace it peeked past.
4. \[/\] and \(/\) are recognized as literal display/ inline math delimiters at the tokenizer level, not as generic control symbols.

Known, deliberate simplifications:

• An unbraced mandatory argument (\section\relax text, vanishingly rare in real manuscripts) reads only a single bare token (one char, or one control-sequence name with no further recursive argument parsing), rather than pylatexenc’s fully general recursive expression read.
• Inside a deliberately unclosed { group (malformed input), this tokenizer keeps tokenizing normally, while real pylatexenc’s tolerant-mode recovery silently drops a stray \end{...} token encountered there — see tests/parser_parity.rs’s KNOWN_DIVERGENCES for the one fixture this affects.

Revisit either only if the fixture-diff harness surfaces a real corpus file that needs it.

Structs

Parser

Functions

parse

Parse source into a top-level node list. Never fails — unclosed groups/environments/math simply extend to end-of-input, matching this crate’s tolerant-only philosophy (the Python side has a separate strict-then-tolerant retry; this port only implements the tolerant behavior, which is what most real documents already resolve to after core/parser.py’s pre-neutralization).