antlr-rust-runtime 0.9.0

High performance Rust runtime and target support for ANTLR v4 generated parsers
Documentation

ANTLR4 Runtime for Rust

Crates.io Version ANTLR Runtime Testsuite

antlr-rust-runtime is a pure Rust runtime and metadata generator for ANTLR v4 lexers and parsers. It is a clean-room implementation written from scratch from the public ANTLR runtime contract; it does not vendor or fork an older Rust ANTLR runtime.

First Steps

1. Install ANTLR4

Follow the ANTLR getting-started guide and install the ANTLR tool jar. The runtime tests currently validate against ANTLR 4.13.2.

2. Install the Rust ANTLR runtime tools

Each ANTLR target language needs a runtime package used by generated parsers. For Rust projects, add the runtime crate:

[dependencies]
antlr-rust-runtime = "0.9"

The library crate is imported as antlr4_runtime:

use antlr4_runtime::{CommonTokenStream, InputStream};

Install the companion generator binary:

cargo install antlr-rust-runtime

This installs antlr4-rust-gen, which turns ANTLR .interp metadata into Rust lexer and parser modules. During generation it also compiles the lexer's DFA ahead of time and embeds the tables in the generated lexer, so tokenization runs at full speed from the first character with no per-process warmup.

3. Generate your parser

The current release uses a metadata-first generation path:

  1. run the official ANTLR tool to produce .interp files,
  2. run antlr4-rust-gen to emit Rust modules,
  3. compile those modules against antlr4_runtime.

For a split lexer/parser grammar:

antlr4 MyGrammarLexer.g4 MyGrammarParser.g4

antlr4-rust-gen \
  --lexer MyGrammarLexer.interp \
  --parser MyGrammarParser.interp \
  --out-dir src/generated

The checked-in ANTLR RustTarget/StringTemplate shell is kept in tool/ and will be expanded around the same runtime contracts.

Alternative: Generate metadata with antlr-ng

antlr-ng is a TypeScript/npm parser generator based on ANTLR 4.13.2. It does not currently ship a Rust target, but it can produce the same .interp metadata that antlr4-rust-gen uses.

Install it with npm or run it through npx:

npx -y antlr-ng -Dlanguage=Java -o build/antlr --exact-output-dir true JSON.g4

The -Dlanguage=Java option selects one of antlr-ng's bundled code-generation targets only so the tool emits grammar artifacts, including JSONLexer.interp and JSON.interp. The Java files can be ignored; Rust code still comes from antlr4-rust-gen:

antlr4-rust-gen \
  --lexer build/antlr/JSONLexer.interp \
  --parser build/antlr/JSON.interp \
  --out-dir src/generated

For local tooling, antlr-ng requires Node.js 20 or newer. See the antlr-ng getting-started guide for CLI installation and option details.

Complete Example

Suppose you are using the JSON grammar from antlr/grammars-v4/json.

Fetch or copy JSON.g4, then generate ANTLR metadata:

antlr4 JSON.g4

Generate Rust modules:

antlr4-rust-gen \
  --lexer JSONLexer.interp \
  --parser JSON.interp \
  --out-dir src/generated

Declare the generated modules in your crate:

mod generated {
    #![allow(dead_code)]

    pub mod json;
    pub mod json_lexer;
}

Call the generated parser helper for the compact path:

use generated::json::{self, Json};
use generated::json_lexer::JsonLexer;

fn main() -> Result<(), antlr4_runtime::AntlrError> {
    let tree = json::parse(r#"{"a":1}"#, JsonLexer::new, Json::json)?;

    println!("{}", tree.text());
    Ok(())
}

Use parse_with_parser when you want the compact setup path and also need the parser afterward for diagnostics or the owned token stream:

use antlr4_runtime::Parser;
use generated::json::{self, Json};
use generated::json_lexer::JsonLexer;

fn main() -> Result<(), antlr4_runtime::AntlrError> {
    let output = json::parse_with_parser(r#"{"a":1}"#, JsonLexer::new, Json::json)?;
    let syntax_errors = output.parser.number_of_syntax_errors();
    let tree = output.result;
    let tokens = output.parser.into_token_stream();

    println!("{} errors across {} tokens", syntax_errors, tokens.tokens().len());
    println!("{}", tree.text());
    Ok(())
}

Or construct each layer explicitly when you need to set source names, parser options, or custom error handling before invoking the entry rule:

use antlr4_runtime::{CommonTokenStream, InputStream};
use generated::json::Json;
use generated::json_lexer::JsonLexer;

fn main() -> Result<(), antlr4_runtime::AntlrError> {
    let lexer = JsonLexer::new(InputStream::new(r#"{"a":1}"#));
    let tokens = CommonTokenStream::new(lexer);
    let mut parser = Json::new(tokens);
    let tree = parser.json()?;

    println!("{}", tree.text());
    Ok(())
}

Choosing Parser Entry Rules

Generated parsers expose one public method per grammar rule. Call the method that matches the grammar's intended top-level rule for the input; the generator can identify rules that are not called by other rules, but it cannot infer the semantic choice between multiple top-level forms. The generated parser rustdoc lists likely entry methods first, followed by all rule methods.

For the JSON grammar above, json() is the natural entry. Larger grammars may have several top-level forms, so confirm the intended entry rule against that grammar's documentation. Calling the wrong rule can still recover and return a parse tree with error nodes, so check parser diagnostics when adding a new input form.

Technical Notes

  • Pure Rust runtime implementation.
  • Written from scratch as a clean-room implementation.
  • Supports ANTLR serialized ATN deserialization.
  • Supports lexer and parser execution through generated Rust wrappers.
  • Supports real split lexer/parser grammars, including Kotlin smoke builds.
  • Passes every upstream ANTLR runtime-testsuite descriptor discovered by the harness: 357 passed, 0 failed, 0 skipped, 357 run.
  • Licensed under BSD-3-Clause for compatibility with ANTLR's runtime licensing pattern and downstream open-source applications.

The runtime contains:

  • IntStream and CharStream
  • UTF-8 input as Unicode scalar values
  • Token, CommonToken, token factories, and TokenSource
  • buffered, channel-aware CommonTokenStream
  • Vocabulary
  • recognizer metadata and error listener plumbing
  • parse tree node types, rule contexts, terminal nodes, error nodes, and walkers
  • ANTLR v4 serialized ATN deserialization
  • lexer ATN recognition with longest-match/rule-priority behavior and lexer actions
  • ahead-of-time compiled lexer DFA tables, built by antlr4-rust-gen and embedded in generated lexers, with per-token escape to ATN interpretation for constructs a finite DFA cannot represent (semantic predicates, recursive lexer rules)
  • parser ATN rule recognition with backtracking over token stream indices
  • antlr4-rust-gen, a Rust generator that consumes ANTLR .interp metadata and emits Rust modules
  • antlr4-runtime-testsuite, a harness for running upstream ANTLR runtime-test descriptors through the Rust metadata path

See docs/kotlin-build.md for the Kotlin smoke workflow. See docs/runtime-testsuite.md for the upstream runtime-testsuite harness.

Semantic Predicates and Actions: the Compatibility Boundary

ANTLR grammars may embed target-language semantic predicates and actions ({isTypeName()}?, {this.count++;}). The serialized ATN records where they occur, but not executable code, so a metadata-first runtime cannot run arbitrary grammar-embedded snippets. The boundary is:

  • Target-agnostic grammars — no embedded code, or only built-in lexer commands (skip, channel(...), mode(...), type(...)) — are fully supported.
  • Recognized predicate/action shapes — a library of common idioms (constant predicates, lookahead text/type checks, integer member counters, column predicates, and the upstream testsuite's action templates) — are translated into SemIR by antlr4-rust-gen when the grammar source is passed via --grammar.
  • User pattern files — --sem-patterns file.toml can add exact predicate rewrites, helper-call rewrites, and per-coordinate hook / assume-true / assume-false / error dispositions without changing the generator.
  • Everything else is not silently guessed. Each generator run writes a semantics.json manifest next to the generated modules listing every predicate/action coordinate with its grammar source span, body, and disposition (translated, hooked, assume-true, assume-false, ignored, synthetic, or error). A synthetic action is one ANTLR inserts itself (e.g. during left-recursion elimination); it has no grammar-author source, is a runtime no-op, and is exempt from the error gate — only actions the author actually wrote in the grammar can fail it.

Unknown coordinates are governed by --sem-unknown:

antlr4-rust-gen --lexer L.interp --parser P.interp --grammar G.g4 \
    --out-dir src/generated --sem-unknown error

  • assume-true (current default, deprecated): unknown predicates pass, unknown actions are no-ops — the historical behavior. A future minor release changes the default to error.

  • hook: unknown parser predicates are routed to SemanticHooks and fail if the hook does not handle them.

  • assume-false: unknown predicates fail, removing the guarded alternatives.

  • error: generation fails, naming each coordinate:

    unsupported semantic predicate: rule=s(0) pred_index=0 at 2:4: {isTypeName()}

At runtime the same policy exists as ParserRuntimeOptions::unknown_predicate_policy (UnknownSemanticPolicy::{AssumeTrue, AssumeFalse, Error}); under Error, evaluating an unknown predicate coordinate fails the parse with AntlrError::Unsupported instead of producing a tree whose shape silently depended on a guess.

Generated parsers also expose a parser-side hook escape hatch: MyParser::with_hooks(tokens, hooks), where hooks implements SemanticHooks. Unknown parser predicates are offered to SemanticHooks::sempred before the fallback policy is applied, and unhandled parser action events are offered to SemanticHooks::action after the committed parse path is selected. Predicate hooks may run speculatively during prediction, so they must be replay-safe.

For bare helper-call predicates, generated parsers also emit a typed hook adapter (MyParserHooks plus MyParserTypedHooks<T>) that maps stable manifest coordinates to named Rust methods. Lexer callers can use LexerSemCtx with atn::lexer::next_token_with_semantic_hooks or the compiled-DFA variant to route lexer predicates/actions through the same SemanticHooks trait.

Generated lexers also own optional hook state and emit typed lexer adapters when a semantic pattern maps lexer helper calls to hooks. The official grammars-v4 JavaScript and TypeScript grammars are complete examples, including checked-in Rust lexer/parser base modules and strict build commands; see docs/javascript-build.md and docs/typescript-build.md.

Use --require-full-semantics in CI when every coordinate must be either translated or explicitly hooked; policy fallbacks fail generation.

Embedded target-language actions are not portable — including in official ANTLR

A grammar that embeds a target-language action (a { ... } block of Java/C#/etc. code, rather than a portable lexer command) is only usable with the language it was written for. This is a limitation of ANTLR itself, not of this runtime: the official ANTLR tool does not translate embedded actions between targets — it copies the source text verbatim into the generated code.

For example, the official Kotlin/kotlin-spec KotlinLexer.g4 contains a Java-only action:

RCURL: '}' { if (!_modeStack.isEmpty()) { popMode(); } };

Generating a Go parser from it with the official tool (antlr4 -Dlanguage=Go KotlinLexer.g4) emits the Java verbatim:

func (l *KotlinLexer) RCURL_Action(localctx antlr.RuleContext, actionIndex int) {
	switch actionIndex {
	case 0:
		if !_modeStack.isEmpty() { // undefined in Go — does not compile
			popMode()              // undefined in Go — does not compile
		}
	}
}

The generated Go fails to compile (undefined: _modeStack, undefined: popMode), and ANTLR offers no supported way to fix it beyond hand-editing the grammar — the grammar even carries a comment telling non-Java users to replace the snippet manually. Every non-Java ANTLR target has this gap.

This runtime does better in two ways:

  1. It recognizes a library of common embedded idioms (e.g. the guarded popMode() above) and maps them to the equivalent portable operation, so many real grammars generate as-is.
  2. For anything it does not recognize, --sem-unknown=error fails loudly at generation time, naming the coordinate, instead of silently emitting uncompilable or no-op code. The fix is to express the action as a portable lexer command (-> popMode, -> pushMode(X), -> type(X), -> channel(HIDDEN)), add a --sem-patterns rewrite, or route it through a SemanticHooks implementation.

Portable lexer commands and the recognized idioms are the target-agnostic subset; prefer them when authoring grammars intended for multiple runtimes.

Grammars whose { ... } blocks are already Rust can skip translation entirely: antlr4-rust-gen --actions embedded --grammar Foo.g4 splices the bodies verbatim (after $-attribute translation) into the generated parser, inline at their ATN action/predicate coordinates. This is the mode the conformance harness uses after rendering descriptor grammars through Rust.test.stg (see below).

Runtime Testsuite

On the maintainer checkout, where the ANTLR jar and upstream runtime-testsuite live under /tmp/antlr-cleanroom, run the full sweep with:

cargo run --release --quiet --bin antlr4-runtime-testsuite

The harness runs descriptors the way every official ANTLR target does: each descriptor grammar is rendered through .conformance-review/Rust.test.stg with the real StringTemplate engine, so its actions and predicates become real Rust code that is compiled and executed inline.

Run a specific descriptor:

cargo run --bin antlr4-runtime-testsuite -- \
  --antlr-jar path/to/antlr-4.13.2-complete.jar \
  --descriptors path/to/antlr4/runtime-testsuite \
  --case LexerExec/KeywordID

Performance

tools/parse-bench/ benchmarks parse throughput of the generated Rust parsers against the upstream Go runtime (github.com/antlr4-go/antlr/v4) — and optionally the reference Python runtime and tree-sitter — on real-world Kotlin, C#, Java, and Trino SQL fixtures. See tools/parse-bench/README.md for setup (the ANTLR jar, the grammars-v4 sparse checkout, and the Python dependencies).

Run the Rust-vs-Go comparison across all fixture languages:

python3 tools/parse-bench/run.py \
  --languages kotlin,csharp,java,trino \
  --runtimes rust-antlr,go-antlr \
  --quick \
  --json target/parse-bench/results.json \
  --markdown target/parse-bench/results.md

The report prints min/avg parse time and a ratio against rust-antlr for every fixture. Drop --quick (or add --iters/--warmups) for longer, lower variance runs; add --runtimes rust-antlr,go-antlr,python-antlr,tree-sitter to include the other runtimes.

Current results

Relative parse speed of this runtime versus the Go runtime, summarized as the geometric mean of the per-fixture go ÷ rust parse-time ratios in each language group (> 1.0 means Rust is faster than Go; < 1.0 means slower):

Language Fixtures Rust vs Go (parse time)
Kotlin 4 ~18× faster
Java 4 ~1.8× faster
C# 4 ~1.2× faster
Trino SQL 5 ~1.1× faster

Rust is faster than Go on every fixture in all four language groups, with Kotlin leading dramatically (expression-ladder memoization in the generated walker). Lexer DFAs are compiled at generation time and embedded in the generated lexer, so tokenization needs no warmup at all; learned parser decision DFAs are shared across parser instances, so repeated parses of the same grammar — the common case for a CLI tool or language server — skip relearning entirely. Numbers are warm-parse minimums on an Apple M3 Pro and are indicative — re-run the benchmark on your own hardware for authoritative figures.

Useful Information