antlr-rust-runtime 0.8.0

High performance Rust runtime and target support for ANTLR v4 generated parsers
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

# Development notes

## Codegen boundaries

Generic runtime and codegen paths must stay grammar-agnostic. Do not add
language-specific rule names, grammar names, file extensions, or semantic
workarounds to `src/bin/antlr4-rust-gen.rs` or runtime modules; model the
behavior from ANTLR metadata/ATN structure instead, and keep language-specific
guidance in docs or tests for that language.

## Kotlin parser parity perf benchmark

Reproduces the timings against the Kotlin grammar from `antlr/grammars-v4`.

### One-time setup (fresh checkout)

```bash
# 1. ANTLR jar (any path; pin v4.13.2)
mkdir -p /tmp/antlr-cleanroom/tools
curl -fLo /tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar \
    https://www.antlr.org/download/antlr-4.13.2-complete.jar

# 2. grammars-v4 checkout (sparse, just the kotlin grammar)
mkdir -p /tmp/antlr-cleanroom/grammars-v4
git -C /tmp/antlr-cleanroom/grammars-v4 init -q
git -C /tmp/antlr-cleanroom/grammars-v4 remote add origin https://github.com/antlr/grammars-v4.git
git -C /tmp/antlr-cleanroom/grammars-v4 sparse-checkout init --cone
git -C /tmp/antlr-cleanroom/grammars-v4 sparse-checkout set kotlin/kotlin
git -C /tmp/antlr-cleanroom/grammars-v4 fetch --depth 1 origin 284602b3f23ca54dc30778204ab7ae9e969145e9
git -C /tmp/antlr-cleanroom/grammars-v4 checkout FETCH_HEAD
```

### Run the parity smoke + dumper build

```bash
tests/kotlin-parity/run.sh \
    --antlr-jar /tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar \
    --grammars-v4 /tmp/antlr-cleanroom/grammars-v4
```

That regenerates the Rust parser from the Kotlin grammar `.interp`, builds `tests/kotlin-parity/dumper`, and asserts the parse trees match `antlr4-python3-runtime` byte-for-byte.

### Measure parse-only timings

The dumper has a built-in parse-only stopwatch so process startup (~10 ms) is excluded:

```bash
DUMPER=tests/kotlin-parity/dumper/target/release/kotlin-parity-dumper
for snippet in tests/kotlin-parity/snippets/*.kt; do
    echo "=== $(basename "$snippet") ==="
    "$DUMPER" --input "$snippet" --output /tmp/dump.txt --iters 5 --time
done
```

`--iters N` repeats parse N times within one process; `--time` prints `min`/`avg` to stderr.

## ANTLR runtime testsuite

Validates the Rust runtime against ANTLR's upstream conformance descriptors.

### One-time setup

```bash
git clone --depth 1 https://github.com/antlr/antlr4 /tmp/antlr-cleanroom/antlr4-upstream
```

The harness reads `antlr4-upstream/runtime-testsuite` and the same ANTLR jar fetched above.

### Run the full sweep

```bash
cargo run --release --quiet --bin antlr4-runtime-testsuite
```

Defaults to `ANTLR4_JAR=/tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar` and `ANTLR4_RUNTIME_TESTSUITE=/tmp/antlr-cleanroom/antlr4-upstream/runtime-testsuite`. Override with `--antlr-jar`/`--descriptors` or env vars. Cases run on `--jobs` parallel workers (default `min(cores, 8)`), each with its own cargo target-dir stripe; the render driver and `antlr4-rust-gen` are prebuilt once per sweep. Wall-clock ≈ 2 minutes on Apple Silicon.

### The rendered (embedded-actions) pipeline

The harness runs descriptors the way every official ANTLR target does:
each descriptor grammar is rendered through
`.conformance-review/Rust.test.stg` with the real StringTemplate engine
(`tools/stg-render/RenderGrammar.java`, executed via the ANTLR jar and the
Java single-file source launcher), so its actions/predicates become real
Rust code. The rendered grammar feeds both the ANTLR tool and
`antlr4-rust-gen --actions embedded`, which splices the bodies verbatim
after `$`-attribute translation (`src/bin_support/embedded.rs`) and
generates typed context views, per-rule attrs structs, members
fields/methods, listener traits, and recognizer facades. `--stg PATH`
overrides the template group. (An earlier template-recognition pipeline,
which simulated action output instead of executing it, was replaced by
this one before ever shipping.)

### Run a subset while iterating

```bash
# One descriptor:
cargo run --release --quiet --bin antlr4-runtime-testsuite -- --case LexerExec/KeywordID

# One group (e.g. while debugging left-recursion):
cargo run --release --quiet --bin antlr4-runtime-testsuite -- --group LeftRecursion --limit 20

# Keep the per-case temp crates for inspection:
cargo run --release --quiet --bin antlr4-runtime-testsuite -- --case ParserErrors/SingleSetInsertion --keep
```

Per-case scratch crates land under `target/antlr-runtime-testsuite/<case>/`. Stale dirs from a killed run can fail a re-run with `Os { code: 66, ... DirectoryNotEmpty }` — `rm -rf target/antlr-runtime-testsuite/*` to recover.

## CI parity

CI runs `cargo clippy --locked --all-targets --all-features -- -D warnings`, so reproduce locally with the same flags before pushing — `clippy::excessive-nesting`, `clippy::disallowed_types`, and similar nursery/pedantic lints all promote to errors there.