# lang-forge v0.2.0 — LexerSketch Forges Languages
**The core, and the hard part of the roadmap.** v0.2.0 turns the scaffold into
LexerSketch: describe a language in a short `.lsf` schematic — its tokens, its
grammar, its operators and their precedence, the capabilities it includes — and
`Language::from_lsf` forges a working front end for it. The forged language
lexes and parses source text into the family's lossless `syntax-lang` tree,
reports every problem as a `diag-lang` diagnostic, never fails and never
panics, and wires the schematic's capabilities into a `pass-lang` pipeline.
There is no code generation step: the language is ready the moment the
schematic is forged. The surface is four types and one alias.
## What is lang-forge?
The capstone of the `-lang` language-construction family. The crates below it
supply the pieces a language is built from — positions, tokens, diagnostics,
syntax trees, passes — and lang-forge assembles them from a description, so a
new language starts as a schematic rather than as a hand-written lexer and
parser. Because its trees are `syntax-lang` trees, a forged language plugs
straight into the formatter, incremental reparser, language server, and
tree-sitter crates of the family. It is the LGEN-tier crate of the family.
## What's new in 0.2.0
### The `.lsf` schematic
A schematic is a NOML document — the static, TOML-compatible core of NOML —
with up to four tables:
```toml
[language]
name = "calc"
extensions = ["calc"]
[lexer]
line_comments = ["#"]
[rules]
program = "stmt*"
[rules.expr]
{ left = ["+", "-"] },
{ left = ["*", "/"] },
{ prefix = ["-"] },
{ right = ["^"] },
]
[capabilities]
include = ["unused-variables"]
```
`[language]` names the language and its file extensions and can pick the
start rule. `[lexer]` chooses Unicode (UAX #31) or ASCII identifiers, makes
line breaks significant, and declares line comments, nesting block comments,
and string forms with their own delimiters and escapes. `[rules]` holds the
grammar in a small rule language — quoted literals, the token classes `IDENT`,
`NUMBER`, `STRING`, and `NEWLINE`, rule references, sequences, ordered choice,
`*`, `+`, `?`, and hidden `_` rules — and expression rules, which list operator
levels from loosest to tightest (`left`, `right`, `none`, `prefix`, `postfix`),
with `then` tails for calls, indexing, and `? :`. `[capabilities]` names the
passes the language includes.
NOML's dynamic features — `env(...)`, `@native` types, includes — are refused,
so a schematic forges the same language on every machine.
### `Language` — forge once, parse anything
```rust
let calc = Language::from_lsf(&std::fs::read_to_string("calc.lsf")?)?;
let parse = calc.parse("let area = 3 * r ^ 2;");
assert!(!parse.has_errors());
```
`from_lsf` (or `str::parse`) reads, checks, and compiles the schematic once;
`parse` then walks precomputed tables, and a `Language` is
`Send` and `Sync`, so one language parses on many threads. `lex` returns the
lossless token stream for syntax highlighting; `kind` and `kind_name` map
between kinds and their names; `name`, `version`, `extensions`, and
`capabilities` expose the schematic's identity.
### A lexer derived from the grammar
The lexer is not written; it is derived. Every literal in the grammar becomes a
keyword (if it reads like an identifier) or a symbol, matched longest first, so
`==` wins over `=` and `//` over `/`. Numbers (`1_000`, `3.25e-4`, `0xFF`,
`0o17`, `0b1010`), identifiers, whitespace, and the declared comments and
strings are built in. Malformed input — unexpected characters, unterminated
strings and comments, invalid digits and suffixes — is reported and kept in the
tree as trivia or as the token it almost was.
### A parser that never fails
Alternatives are tried in order and the first that matches wins, as in a
parsing expression grammar. FIRST sets prune that search before it starts, so
almost every decision has exactly one candidate and the parser commits at once.
It speculates only where alternatives genuinely share a first token, and
memoizes every speculative attempt by rule and position, so alternatives that
share a prefix through different rules parse it once and nested input stays
linear. Alternatives that begin alike are left-factored when the language is
forged.
Expression rules parse by precedence climbing, with operator nodes spliced
around their left operands by a forward link instead of an insertion.
Recovery is part of the design, not an afterthought. A missing token is
reported and assumed; a token no rule wants is reported and wrapped in an
`ERROR` node; skipping stops at any token an enclosing construct is waiting for
— the `;` that ends the statement, the `}` that closes the block — so one
mistake does not swallow the rest of the file; and at most one syntax error is
reported per token:
```text
error: expected `)`, found `;`
--> sample.mini:5:21
|
```
Every recursive step of the parser counts against a limit of 768 grammar
levels — at most about 256 KiB of stack in a release build and 768 KiB in a
debug build, measured on the deepest-nesting grammars known — so input nested
beyond it is reported rather than followed and no input can exhaust the stack.
Every tree is lossless: it covers every byte of the source. With
`newlines = true`, the end of the input counts as a line break wherever a
`NEWLINE` may come, so the last line of a file needs none.
### Forge-time checks that name the fix
Forging reports everything wrong with a schematic at once, in source order,
each problem pointing into the schematic: NOML syntax errors; unknown, missing,
and mistyped settings; malformed rules; undefined rules, with a suggestion;
literals the lexer cannot produce; delimiters used for two things; left
recursion, named as a chain; repetitions of something that can match nothing;
and alternatives that can never match because an earlier one always wins.
```text
error: undefined rule `statment`
help: did you mean `statement`?
error: rule `expr` is left-recursive: expr → expr
help: a rule cannot begin with itself; write lists as `item (',' item)*`
and operators as an expression rule ([rules.name] with `levels`)
```
### Capabilities
LexerSketch never bakes a language feature in. A schematic names the
capabilities it includes; `Language::pipeline` takes a registry of
`pass-lang` passes — which may serve many languages — and assembles the ones
the schematic names, in schematic order, into a `PassManager`, refusing if one
is missing or ambiguous. A pass reads the `Parse` it is given and reports with
`Parse::report`.
```rust
let registry: Vec<Capability> = vec![Box::new(UnusedVariables), Box::new(Formatting)];
let mut pipeline = mini.pipeline(registry)?;
pipeline.run(&mut parse)?;
```
### Examples
Five runnable examples and four schematics ship. `calc` forges a calculator and
evaluates programs by walking the tree. `check` is a command-line checker for
any schematic and source file, with rendered diagnostics and an optional tree
dump. `json` validates JSON documents. `capabilities` wires an
`unused-variables` pass into `mini`. `highlight` paints source by token kind
from `Language::lex`. The schematics — `calc.lsf`, `json.lsf`, `mini.lsf` (a
small imperative language with eight precedence levels), and `conf.lsf` (an
INI-style format with significant line breaks) — double as test fixtures.
### Tests and benchmarks
Every public item has rustdoc with runnable examples, and every `rust` example
in `README.md` and `docs/API.md` runs as a doctest. A strict reference parser
in the unit tests holds the recovering parser to the grammar: whenever the
recovering parser reports no error, the two trees must be identical. Property
tests generate random valid programs, which must parse without a diagnostic;
throw token soup and arbitrary Unicode at every example language, whose trees
must be lossless and properly nested; forge arbitrary and randomly mutated
schematics, which must succeed or fail cleanly; and parse with memoization on
and off, which must give identical trees and diagnostics.
### Adversarial review
Before this release, an independent review attacked every guarantee above
with hand-built schematics and twenty thousand random grammars. Trees stayed
lossless, the strict and recovering parsers never disagreed, and left-factoring
never misreported an alternative. It did break seven things, all fixed here and
each kept as a regression test in `tests/regressions.rs`:
- The depth limit counted rules but not the nesting inside them, so a rule of
nested optionals could overflow the stack. Now every recursive step counts.
- Alternatives sharing a prefix through different rules re-parsed it at every
level of nesting: exponential time on inputs under 100 bytes. Speculative
attempts are now memoized, and on input already too deep to parse,
speculation stops at the first failure.
- When every alternative failed, re-running the best one repeated the
speculation beneath it: quadratic time. The memo removes that too.
- A long dotted key in a schematic could overflow the stack while forging.
Dotted keys and headers are now limited to 64 parts, like other nesting.
- Recovery that assumed a missing token could re-enter a rule at the same
position until the depth limit. A rule is now never re-entered where it is
already active without consuming input.
- The end of the input matched `NEWLINE` only as a bare token, not through a
choice, a repetition, or a rule. It now does everywhere.
- An invalid escape before a multi-byte character produced an error span that
split the character. It now covers the whole character.
The review also measured forging cost on extreme schematics. A chain of 16,000
rules took 2.5 s and 30,000 alternatives took 1.5 GB of memory. The analysis
now visits rules in dependency order and allocates token sets only where they
are read: the chain forges in under a second even in a debug build, and the
alternatives in bounded memory.
### Dependency wiring
`syntax-lang` 1, `diag-lang` 1, `pass-lang` 1, and `unicode-lang` 1 are wired,
and the first three are re-exported. `grammar-lang` is not wired: it is still
an unpublished scaffold, so the grammar engine lives in lang-forge as private
modules that can move to `grammar-lang` later without an API change.
`lexer-lang`, `parser-lang`, and `ast-lang` are not wired either, for reasons
recorded in `dev/ROADMAP.md` under the anti-deferral rule; nor is the `noml`
crate, whose runtime dependencies a schematic reader does not need. The
scaffold's unused `serde` feature and `loom` dev-dependency were removed.
### Scaffold fixes
`Cargo.toml` had unquoted `keywords` and `categories`, so the manifest did not
parse at all; `clippy.toml` declared MSRV 1.87 against the crate's 1.85;
`deny.toml` named another project; two docs carried byte-order marks; the
README linked a `dev/DIRECTIVES.md` that does not exist; and the crate root
lacked the REPS lint set. All are fixed, and CI now also runs clippy and the
tests without default features and builds the examples.
## Breaking changes
**None.** v0.1.0 exposed no public API; everything in 0.2.0 is new surface.
The removed `serde` feature had no code behind it.
## Verification
Run on Windows x86_64 and Linux (WSL2 Ubuntu, from a sibling-free copy, as CI
checks the crate out), Rust stable and the 1.85 MSRV; the same commands run in
the configured CI matrix:
```bash
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo clippy --all-targets --no-default-features -- -D warnings
cargo test
cargo test --all-features
cargo test --no-default-features
cargo build --examples
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo +1.85 test --all-features
cargo +1.85 clippy --all-targets --all-features -- -D warnings
cargo +1.85 clippy --all-targets --no-default-features -- -D warnings
cargo deny check
cargo audit
cargo publish --dry-run
```
All green. Counts at this tag (identical for every feature combination):
- 71 unit + 39 integration (forging) + 20 integration (parsing) + 5 property +
13 regression + 64 doctests. The doctests include every `rust` example in `README.md` and
`docs/API.md`, wired into `cargo test` so the published examples cannot
drift from the API.
The property tests were also soaked at 10,000 to 20,000 cases each.
Benchmark baselines (x86_64, release profile; parse figures include lexing,
building, and dropping the tree):
| `forge/mini` | ~38 µs | ~23 µs |
| `forge/json` | ~15 µs | ~9 µs |
| `lex/mini/1MB` | ~3.7 ms | ~1.8 ms |
| `lex/json/1MB` | ~3.9 ms | ~2.1 ms |
| `parse/mini/4KB` | ~94 µs | ~72 µs |
| `parse/mini/1MB` | ~57 ms | ~24 ms |
| `parse/json/1MB` | ~60 ms | ~34 ms |
| `parse/speculative/256KB` | ~33 ms | ~13 ms |
| `parse/calc/chain_100k` | ~37 ms | ~14 ms |
| `parse/mini/errors` | ~25 ms | ~8.8 ms |
On large inputs the tree dominates: on Linux, for 1 MB of mini code, lexing
takes about 2 ms and the parser about 3 ms, while allocating the tree's nodes
takes about 6 ms and freeing them about 13 ms.
## What's next
- **1.0.0 — API freeze.** Freeze the 0.2.0 surface as the 1.0 contract, mark
`docs/API.md` stable with a SemVer promise, and confirm the full test and
benchmark suite on all three platforms. Tree-sitter grammar emission, named
token classes, contextual keywords, and a reusable parse session are
additive later-minor work, not 1.0 requirements.
## Installation
```toml
[dependencies]
lang-forge = "0.2"
```
MSRV: Rust 1.85.
## Documentation
- [README](https://github.com/jamesgober/lang-forge/blob/main/README.md)
- [API Reference](https://github.com/jamesgober/lang-forge/blob/main/docs/API.md)
- [CHANGELOG](https://github.com/jamesgober/lang-forge/blob/main/CHANGELOG.md)
---
**Full diff:** [`v0.1.0...v0.2.0`](https://github.com/jamesgober/lang-forge/compare/v0.1.0...v0.2.0).
**Changelog:** [`CHANGELOG.md`](https://github.com/jamesgober/lang-forge/blob/main/CHANGELOG.md#020---2026-10-07).