████████╗███████╗██╗ ██╗██████╗ ███████╗
╚══██╔══╝██╔════╝╚██╗██╔╝██╔══██╗██╔════╝
██║ █████╗ ╚███╔╝ ██████╔╝███████╗
██║ ██╔══╝ ██╔██╗ ██╔══██╗╚════██║
██║ ███████╗██╔╝ ██╗██║ ██║███████║
╚═╝ ╚══════╝╚═╝ ╚═╝╚═╝ ╚═╝╚══════╝
[TEX'S MOUTH AND EXPANDER, COMPILED TO BYTECODE — NOT INTERPRETED]
"Every TeX since 1982 interprets the expander. This one compiles it."
A TeX engine in Rust: Knuth's mouth and expander, lowered onto
fusevm bytecode and run on the
shared three-tier Cranelift JIT — the same engine behind zshrs, stryke,
rubylang, pythonrs and scalars.
Table of Contents
- [0x00] What it is
- [0x01] Install
- [0x02] Usage
- [0x03] What works
- [0x04] Intercepts
- [0x05] Inline Rust
- [0x06] What does not
- [0x07] How it runs
- [0x08] Parity
- [0x09] Fuzzing
- [0x0A] Benchmarks
- [0x0B] Documentation
- [0xFF] Licence
A TeX engine in Rust: Knuth's mouth and expander, built to be lowered onto a bytecode VM.
[0x00] What it is
TeX is two machines. The mouth turns bytes into tokens under a mutable
category-code table; the expander turns tokens into other tokens — \def,
\csname, \the, the conditionals. Only after that does the stomach build
boxes and ship DVI.
texrs implements the first two. That is the half a macro-heavy document spends
its time in, and the half where a compiled implementation has something to prove:
every mainstream engine (pdfTeX, XeTeX, LuaTeX) descends from tex.web through
web2c and interprets the expander.
[0x01] Install
# Homebrew (macOS + Linux)
# Or via crates.io
# Or from source
&& &&
Zsh tab completion
Editors
texrs --lsp and texrs --dap are ordinary LSP/DAP servers over stdio, so any
client can drive them. editors/ carries ready-made configuration:
Man pages
[0x02] Usage
texrs takes tex's own command line, in all three of its invocation forms:
A bare name gets .tex appended, so texrs doc and texrs doc.tex are the
same run. Options may be spelled with one dash or two, and a value may follow
an = or a space — -jobname=x, --jobname=x and -jobname x all agree.
tex's options:
-interaction=MODE
-jobname=NAME
-output-directory=DIR
-progname=NAME
-fmt=NAME
texrs's own:
Two places the grammar departs from tex, both because texrs takes several
files where tex takes one: a non-option argument is a FILE unless it begins
with \, and options are recognised anywhere rather than only before the
first file, so texrs doc -halt-on-error sets the flag instead of typesetting
it.
Output is written the way tex writes it on the terminal, which is what the parity harnesses compare:
$ texrs examples/macros.tex
(./examples/macros.tex HELLO-WORLD [1|2] )
examples/ carries a runnable program per construct — macros with delimited
parameters, count arithmetic, conditionals, groups, \csname, \edef — and
tests/examples.rs holds every one of them in parity with real tex, with no
known-gap escape hatch. Documentation that has drifted from the engine is worse
than none.
[0x03] What works
- Category codes,
\catcode, and INITEX's sparse defaults —{is not a group character until something makes it one, exactly as a baretexrun behaves. - The three-state line scanner: blank line to
\par, spaces collapsed, the space after a control word swallowed and after a control symbol kept. ^^Xnotation.\defwith undelimited and delimited parameters (\def\pair#1,#2.{...}),##, and nested definitions.\csname/\endcsname,\string,\the,\number,\expandafter.\let,\edef/\xdef,\gdef,\global,\begingroup/\endgroup.- Conditionals:
\iftrue,\iffalse,\ifnum,\ifodd,\ifx,\ifcasewith\or,\else,\fi— nested, and inside a\messagebody. - Groups, which scope the macro table AND the count registers they write.
\countregisters,`xcharacter codes,\advance/\multiply/\divide.\message.
[0x04] Intercepts
\intercept % => [in]HELLO-WORLD
\intercept % every sectioning macro, including
% the ones a package defines later
\intercept % => <<HELLO-WORLD>>
Advice on macro expansion — before, after, around, with \proceed
standing for the original expansion inside an around handler. The pattern is a
glob over macro names, which is what makes it useful on a macro package: the
advice is registered before the macros it will catch exist.
Expansion is a compile-time act here, so advice is woven into the token stream and is undone by the group that registered it, like any other assignment. A handler that calls the macro it advises does not weave itself — a call inside advice is not advised.
[0x05] Inline Rust
\rust
\count1=21
\message % => 42
The block is compiled by rustc, loaded, and its exported functions become
callable. Its body is Rust, not TeX, so it is lifted out of the file before
the mouth reads it — #, {, } and & are category codes the mouth would
act on. A call is a number wherever TeX reads one: a register assignment, an
arithmetic operand, a conditional, or a \message body.
The compiled library is cached by body hash, so only the first run pays for the compile, and a block that does not compile stops the run with rustc's own diagnostic rather than a missing-function error later.
[0x06] What does not
No boxes, no glue, no paragraph breaking, no fonts, no DVI. This is not a
typesetter yet — see docs/ROADMAP.md.
[0x07] How it runs
texrs is a fusevm frontend, not an interpreter: mouth → expander → command
stream → fusevm bytecode → the VM runs it. A count register is a VM slot, so
\advance\count0 by 5 is GetSlot / LoadInt / Add / SetSlot — native ops the
JIT can compile. \ifnum is NumGt + JumpIfFalse, a real branch.
A conditional whose truth depends only on the macro table (\iftrue, \ifx)
is folded while lowering instead, because there is nothing for the VM to test.
tests/lowering.rs asserts the emitted bytecode rather than the printed output —
output parity alone would not distinguish a frontend from a tree-walker.
[0x08] Parity
The contract is the \message stream, compared byte-for-byte against the real
tex binary. No expectation is written by hand:
Both harnesses read the engine version they were measured against out of
BUGS.md and refuse to run against a different tex: a mismatched oracle does
not fail loudly, it reports a different set of divergences, which reads exactly
like a regression.
The corpus is small and deliberately awkward — ## inside a nested \def,
catcode changes mid-file, \csname built from a macro, control-word space
swallowing, conditionals nested inside a \message body, \ifcase,
\expandafter, \edef freezing a register, a \count assignment scoped by a
group. Every case is in parity except the ones tests/known_gaps.txt names,
and the gate fails both on an unlisted divergence and on a listed case that has
started passing, so the list cannot go stale.
[0x09] Fuzzing
Hand-written cases only cover what someone thought to write down.
scripts/fuzz_parity.sh generates seeded random programs confined to the
implemented subset, runs them under both engines in parallel, and REDUCES
whatever diverges — dropping statements for as long as the divergence survives,
and refusing a reduction that changes the divergence into a different one — so
what it hands back is small enough to commit to tests/cases. The same seed
generates the same corpus on any machine.
fuzz/ is a cargo-fuzz crate (targets lex, lower, run) looking for panics
rather than divergences. tests/fuzz_smoke.rs replays each target on its seed
corpus under stable Rust, and tests/fuzz_mass_replay.rs points the mouth at
generated mutations of every .tex in the tree, so cargo test still exercises
the harness on a machine with no nightly toolchain.
[0x0A] Benchmarks
cargo bench separates the stages, because one end-to-end number cannot tell a
slow expander from a slow VM, and sweeps document size, because a frontend
that is quadratic in the number of macro calls looks fine on a small file and
stops being usable on a real one. That sweep has already earned its keep: it
caught a line-number lookup that was O(n) per token, and fixing it took an
800-statement document from 248 ms to 11.7 ms.
bench/compare.sh is the comparison that says whether the engine is fast
rather than where its time goes, and prints the two caveats with the numbers:
tex loads the plain format on every run while texrs loads nothing, and texrs
implements the mouth and expander only.
[0x0B] Documentation
- Docs hub — menketechnologies.github.io/texrs (
docs/index.html) - Engineering report — architecture, what lowering forces, parity posture, dependencies (
docs/report.html) - Primitive reference — every primitive texrs carries and where it happens (
docs/reference.html, generated fromsrc/corpus.rswithcargo run --bin gen-docs) - Known gaps — the ledger, each entry pinned by a case the suite gates on (
BUGS.md) - Roadmap — what the stomach would take (
docs/ROADMAP.md)
[0xFF] Licence
MIT.