# ANTLR Runtime Testsuite
ANTLR maintains a shared runtime conformance suite in `antlr/antlr4/runtime-testsuite`.
This repo includes `antlr4-runtime-testsuite`, a Rust-side harness that consumes those
upstream descriptor files without vendoring them.
## Why a Rust Harness Exists
The upstream Java/JUnit harness assumes each target can be generated directly with
`-Dlanguage=<target>` and that target-specific grammar action templates are available.
This runtime currently uses a clean-room metadata path:
1. the official ANTLR tool emits `.interp` metadata,
2. `antlr4-rust-gen` emits Rust modules from that metadata,
3. the generated modules run against `antlr4_runtime`.
The harness follows that path while still using the upstream descriptor grammar,
input, stdout, and stderr expectations.
## Run Full Sweep
On the maintainer checkout, where the ANTLR jar and upstream runtime-testsuite
live under `/tmp/antlr-cleanroom`, the full Rust sweep is:
```bash
cargo run --quiet --bin antlr4-runtime-testsuite
```
In other environments, pass explicit paths or set `ANTLR4_JAR` and
`ANTLR4_RUNTIME_TESTSUITE`.
## Run One Descriptor
```bash
cargo run --bin antlr4-runtime-testsuite -- \
--antlr-jar /tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar \
--descriptors /tmp/antlr-cleanroom/antlr4-upstream/runtime-testsuite \
--case LexerExec/KeywordID
```
`--descriptors` may point either at the upstream `runtime-testsuite` directory or
directly at its `resources/org/antlr/v4/test/runtime/descriptors` directory.
## Run a Group Sample
```bash
cargo run --bin antlr4-runtime-testsuite -- \
--antlr-jar /tmp/antlr-cleanroom/tools/antlr-4.13.2-complete.jar \
--descriptors /tmp/antlr-cleanroom/antlr4-upstream/runtime-testsuite \
--group LexerExec \
--limit 20
```
The harness creates temporary Cargo crates under `target/antlr-runtime-testsuite`.
Pass `--keep` to retain those directories for debugging.
## Current Scope
Supported now:
- lexer descriptors,
- parser descriptors with empty stdout/stderr expectations,
- single-grammar descriptors,
- descriptor stdout/stderr comparison,
- grouped lexer recovery diagnostics,
- farthest-token parser mismatch diagnostics for supported non-recovery
failures,
- parser single-token insertion/deletion recovery diagnostics for supported
descriptors,
- parser mismatched-token recovery diagnostics and error-node parse trees for
supported descriptors,
- parser extraneous-input diagnostics and error-node parse trees for supported
single-token deletion descriptors,
- parser precedence predicates in metadata-driven recognition,
- lexer and parser target-template actions for the currently supported stdout
helpers,
- parser token-label text actions such as `$TOKEN.text` and `$label.text`,
- parser `AppendStr(..., "$TOKEN.text")` stdout actions for supported
semantic-predicate descriptors,
- parser token-display actions such as `Append(..., "$label")` and
`Append(..., "$rule.stop")` for recovered-token descriptors,
- parser rule-level `@after` actions for the currently supported stdout helpers,
- parser `$text` action intervals that stop at the previous visible token,
including the greedy and non-greedy if/else binding descriptors,
- parser decision-order tie breaking for clean action-bearing ambiguities such
as optional `else` binding and assignment-vs-wildcard alternatives,
- parser rule-level `@init {<GetExpectedTokenNames():writeln()>}` actions,
- nested parser tree construction for action-bearing rules and direct
`ToStringTree("$ctx")` stdout actions,
- lexer semantic predicates for the currently supported `True()`, `False()`,
`TextEquals(...)`, token-start-column, and current-column templates,
- lexer DFA dump output for the predicate-sensitive `SemPredEvalLexer`
`showDFA` fixtures,
- lexer accept-position adjustment for the upstream `PositionAdjustingLexer`
target template,
- parser `@init {<BuildParseTrees()>}` and `notBuildParseTree` descriptors,
- parser `predictionMode=LL` and `predictionMode=SLL` descriptors modeled by
the metadata recognizer,
- parser `showDiagnosticErrors` ambiguity diagnostics for the currently modeled
exact-ambiguity semantic-predicate descriptors,
- parser `DumpDFA()` output for the currently modeled full-context diagnostics
descriptors,
- parser rule-level `@after {<ToStringTree("$label.ctx")>}` actions for simple
rule labels,
- parser semantic predicates for `LANotEquals(...)` and `LTEquals(...)`
lookahead target templates,
- parser rule-argument predicates for supported `ValEquals("$i", "...")`
target templates, including literal integer calls and `VarRef("i")`
forwarding,
- parser boolean-member predicates for the runtime-testsuite
`GetMember(...):Not()` fixture,
- parser integer-member target templates for semantic-predicate fixtures,
including `AddMember`, `GetMember`, `ModMemberEquals`, and
`ModMemberNotEquals`,
- multi-template parser action blocks and empty regular actions that must stay
aligned with serialized ATN action states,
- parser supported-predicate decision ordering for action-bearing alternatives,
- listener-suite target templates for `BasicListener`, token/rule getter
listeners, and the left-recursive listener fixtures,
- simple left-recursive return-value stdout helpers such as `$e.v` and
`$e.result`,
- common-label left-recursion compile-check templates such as `Production(...)`
and `Result(...)`,
- integer/member predicate scaffolding used by selected semantic-predicate
descriptors, including `InitIntMember`, `SetMember`, and `Invoke_pred`,
- alt-numbered parse-tree contexts for grammars using
`TreeNodeWithAltNumField`/`contextSuperClass`,
- `RuleInvocationStack()` stdout helper actions,
- `BailErrorStrategy()` descriptors as no-ops while the default Rust error
handling still matches the covered outputs,
- compile-time-only target templates such as `IntArg`, `AssignLocal`,
`AssertIsList`, `Pass`, parser property helpers, and supported member
scaffolding as no-ops,
- nested `StringTemplate` action parsing for supported no-op wrappers,
- `StringTemplate` comments in descriptor grammars,
- ANTLR recursive-context tree rewrites for left-recursive parse-tree output,
- ANTLR whitespace escaping for terminal text in `ToStringTree(...)` output,
- `StringTemplate` backslash rendering for descriptor grammars,
- official ANTLR `.interp` generation,
- Rust module generation and execution through Cargo.
Not wired yet:
- composite grammar override/member/mixed-action shapes beyond the currently
supported import metadata cases,
- target-template semantic actions beyond the currently supported stdout helpers
and no-op compile checks,
- parser error recovery diagnostics beyond the currently supported mismatch,
no-viable, extraneous-input, semantic-predicate fail options, EOF unwind, and
token recovery cases,
- runtime diagnostic/profile/DFA flags beyond the currently modeled ambiguity
diagnostics and non-default prediction modes.
The harness reports unsupported descriptors as skipped and treats output mismatches
as failures.
Current validated groups:
- full descriptor sweep: `357 passed, 0 failed, 0 skipped, 357 run`
- `CompositeLexers`: `2 passed, 0 failed, 0 skipped, 2 run`
- `CompositeParsers`: `15 passed, 0 failed, 0 skipped, 15 run`
- `FullContextParsing`: `15 passed, 0 failed, 0 skipped, 15 run`
- `LexerExec`: `42 passed, 0 failed, 0 skipped, 42 run`
- `LexerErrors`: `12 passed, 0 failed, 0 skipped, 12 run`
- `LeftRecursion`: `98 passed, 0 failed, 0 skipped, 98 run`
- `Listeners`: `7 passed, 0 failed, 0 skipped, 7 run`
- `ParseTrees`: `10 passed, 0 failed, 0 skipped, 10 run`
- `ParserExec`: `50 passed, 0 failed, 0 skipped, 50 run`
- `ParserErrors`: `34 passed, 0 failed, 0 skipped, 34 run`
- `Performance`: `7 passed, 0 failed, 0 skipped, 7 run`
- `SemPredEvalLexer`: `8 passed, 0 failed, 0 skipped, 8 run`
- `SemPredEvalParser`: `26 passed, 0 failed, 0 skipped, 26 run`
- `Sets`: `31 passed, 0 failed, 0 skipped, 31 run`