Expand description
§tabnas for Rust
Rust port of the grammar-free Tabnas parser engine. TypeScript is the canonical
implementation; the shared fixtures in ../test/spec define cross-port
behavior.
The crates.io package is tabnas-parser. The Rust library name remains
tabnas:
cargo add tabnas-parser --rename tabnasuse tabnas::{Tabnas, Value};
let parser = Tabnas::new();
// Install rules and actions, then parse input with parser.parse(...).Serialized function references are bound through typed Rust registrations before the grammar is installed:
use tabnas::Tabnas;
let mut parser = Tabnas::new();
parser.alt_condition("@positive", |rule, _ctx| {
rule.o0().is_some_and(|token| token.src != "0")
});
parser.grammar_json(
r##"{"rule":{"top":{"open":[{"s":"#NR","c":"@positive"}]}}}"##,
)?;Typed registrations cover conditions (c), modifiers (h), errors (e),
and dynamic push, replace, and backtrack fields (p, r, b). Modifiers use
an explicit take-and-return contract, and missing or mistyped references fail
transactionally during grammar installation.
The complete condition tier receives the shared live AltMatch and lexer in
one callback, so plugins can perform canonical arbitrary-depth lookahead
without losing match-state mutations.
Reserved lifecycle references (@rule-bo, -ao, -bc, -ac, with
/prepend, /append, and /replace) bind through state_action_ref and are
wired when that serialized rule is installed.
Function token matchers use match_token_ref. They receive the remaining
source and return a non-empty owned source prefix plus its value. This
effect-based interface supports eager and parser-slot-gated matchers without
giving callbacks mutable access to the lexer cursor. Native plugins that need
the mature callback surface use imperative_lex_match_ref; those callbacks
receive the live lexer, rule, and context and may construct native tokens.
High-priority options.match.value entries support serialized regexps and
typed match_value_ref callbacks. Regexp entries can bind their capture
transform with value_transform_ref; function entries return the owned source
prefix and #VL value directly.
Regexp-backed value definitions can bind a named transformer with
value_transform_ref. It receives the whole match followed by capture groups
and returns the token value without direct cursor access.
Unquoted-text modifier pipelines use text_modifier_ref and retain serialized
declaration order. imperative_text_modifier_ref exposes the live lexer, rule,
context, and resolved options.
Matcher-family check hooks use lex_check_ref; callbacks can continue,
skip that matcher, or emit an owned non-empty prefix token.
Priority-ordered custom lexer entries bind through lex_match_ref and are
interleaved with the eight built-in matcher bands. During negotiated lexing,
unwanted custom token effects are discarded without moving the cursor.
Custom token identities can be allocated explicitly with Tabnas::token, or
implicitly by a serialized rule/token-set reference; LexCheckToken::named
resolves the numeric identity after grammar installation.
Function-form comment terminators use comment_suffix_ref and likewise
consume only a validated owned source prefix.
Serialized error, hint, errmsg, and color overlays configure both
structured diagnostic text and human-readable error rendering, including
source context, links, and internal suffix suppression or replacement.
Dynamic suffix references bind through error_suffix_ref and receive an
owned snapshot of the rendered diagnostic and its color palette.
Named parse lifecycle hooks bind through parse_prepare_ref and
parse_budget_ref; serialized prepare maps are deterministic by callback name
and budget callbacks remain inactive until checkEveryN is non-zero.
Parse guards, installed by name through parse_guard, are Rust’s own and
have no TypeScript or Go counterpart. The loop runs each one at every step,
ahead of the budget, and a guard that returns false stops the parse with
cancel. A caller’s parse_budget replaces the budget and leaves the
guards alone, so a grammar bounds nesting with one: a Value drops and
displays by recursion, and a deep one can overflow a small stack.
Load-time options.config.modify callbacks bind through
config_modifier_ref, run in declaration order after option resolution, and
are reapplied on later grammar option overlays until removed.
config_modifier_with_options_ref additionally exposes the immutable raw
option input for the configure pass. Raw options and resolved configuration are
kept separate so non-idempotent modifiers do not compound during overlays or
derivation.
Replacement options.parser.start entry points bind through typed simple,
instance-aware, and parent-context-aware registrations; they bypass the rule
engine like the TypeScript/Go option, and callback panics are returned as
structured internal errors.
Native plugins, grammar-wide group settings, plugin option bags, derived
instances, empty, and commutative instance merge are also available as
typed Rust APIs. Lazy token values, full pre-parse hooks, live lexer checks,
function comment suffixes, and setup-time matcher factories preserve their
canonical callback timing.
The strict-JSON grammar exposed by Tabnas::make_json() is a compatibility
fixture while the Rust plugin API is stabilized. It is not a built-in default
grammar: Tabnas::new() has no rules.
§Parity status
Implemented: ordered grammar rules, open/close/push/replace transitions,
inheritable n and k state, rule-local u state, named actions, built-in
lexers, Unicode-scalar source positions, structured diagnostics, and the shared
strict-JSON, lexer, diagnostic, utility, and divergence-register fixtures.
Serialized grammars also support declarative conditions, group filters,
load-bound builtin configuration, tree/value builtins, token and rule
subscribers, public parse actions with bounded mark/rewind history, parse
budgets, path-aware continuation queries, and opt-in panic-mode recovery with
structured recovery diagnostics. Opt-in negotiated lexing can re-cut contested
source spans for scannerless serialized grammars, including rollback when a
candidate alternate later fails. Typed named hooks cover alternate conditions,
modifiers, errors, dynamic routing/backtracking, lifecycle state, and
effect-based or fully imperative token matchers. The live native surface
includes the resolved AltMatch, next-rule snapshots, live lexer access,
rule-spec inspection, per-rule lifecycle gates, parent-context seeding, token
detail bags, ignored trivia, negotiated re-lex checkpoints, and ordered
subscriber events. Callback panics become structured internal diagnostics at
public parser and lexer boundaries, retaining the active rule, complete rule
stack, token, and source position when a parse context exists. Debug
configuration, stable value formatting, instance descriptions, and opt-in
lexer/rule tracing are built in. Opt-in typed MapRef, ListRef, and Text
results expose parse metadata while serializing to the same plain JSON shape as
TypeScript. Rust directly executes every non-exempt shared TSV fixture; the
fixture paths are not copied into the crate. The repository gate requires every
non-exempt shared fixture to have a TypeScript, Go, and Rust runner, and the
strict-JSON differential gate compares token streams over every data row in
both JSON and parser corpora. Additional compiler-consumer gates compare Rust
against TypeScript over pure-data grammars emitted by the current ABNF, EBNF,
and GBNF compilers.
A slot that names a token set (#KEY, #VAL, a custom set) keeps its
names on the alternate and is resolved again against the instance’s
options whenever the parser is rebuilt, so a set overridden after the
alternate was installed reaches it, as in TypeScript and Go
(tabnas/parser#217).
Cost, not only answers, is tracked against the canonical runtime. A rule
with N elements is flattened by the tree builders once per level of the
repetition that produced it, in every runtime: TypeScript spreads each
level’s kids into the level above, and so does this crate, so both curves
are quadratic in the element count of one rule. A level whose own kids
are still empty takes the array of the level below by handle rather than
cloning it element by element. Measured on list = item *( "," item )
compiled with builtins, release profile, 4000 and 8000 items: TypeScript
0.38 s and 2.1 s, this crate 0.7 s and 6.8 s. What remains is the
algorithm the canonical runtime also runs, at this crate’s per-element
cost for a refcounted value.
Two answers are deliberately this runtime’s own, and this crate’s own
suite pins both. Object key order is out of the parsed-value contract:
Value::Object keeps insertion order,
TypeScript’s plain object puts integer-like keys first, and this crate must
never emulate that. A parse that sets no value answers Value::Null, where
TypeScript answers undefined; the engine’s Value::Undefined is unwrapped
at the parse boundary on purpose.
Binary formats parse here, with three costs the other runtimes do not
pay. Lexer.src is a &str, so it must hold valid UTF-8 and arbitrary
bytes are not: map each byte to the char with that code point
(U+0000..U+00FF) on the way in, which is lossless and makes the
Unicode-scalar index equal to the byte offset. Read a position off
site.pos and never off site.si, which is a UTF-8 byte offset into
the transcoded source and runs ahead at the first byte above 0x7F. The
memory cost of holding bytes this way is real: the transcoded &str is
up to twice the input, and the lexer additionally materialises a
Vec<char> and a Vec<usize> over it.
The second cost is the one that shapes a grammar. match_tokens
callbacks are gated on the rule’s expected-token column exactly as in
TypeScript and Go, but their signature is Fn(&str), so they cannot see
the rule. A field whose length was read from an earlier field therefore
cannot use that path; it has to be an ImperativeLexMatcher under
lex.matchers, which does receive &mut Rule but is NOT column-gated,
so the matcher gates itself on the rule that wants it. Giving the
column-gated callback a rule-aware form would remove the split.
The third is that Token.src is an owned TokenText. TypeScript can
defer a payload to a bare (sI, len) span and Go can slice its source
string without copying; here a payload token copies its bytes.
tests/binary_grammar_test.rs is the worked example, mirroring
ts/test/binary-grammar.test.js and go/binarygrammar_test.go over
byte-identical input.
The portable serialized contract and native imperative tier have been audited
against the TypeScript and Go surfaces, and the audit is executable rather
than asserted: the fixture registration gate, the token-stream differential
and the compiler-consumer gates are what hold it, and what the prose claims
is what those gates run. The claim is not that no difference remains.
rule.child is the rule its parent pushed, name and node together, and
its immediate next is the replacement that took its place, both as in
TypeScript and Go. Walking rule.child.next.next and beyond reaches the
rest of a replacement chain in TypeScript and Go and reaches nothing
here. The shared fixture test/spec/divergent.tsv registers that split,
with a control row, as “Forward traversal of a replacement chain in
Rust”. Known remaining API differences are listed under “Gaps against
the canonical surface” below.
Rust ownership is expressed explicitly:
grammar and next-rule views are immutable snapshots, live mutation is limited
to the &mut Rule, &mut Context, &mut Lexer, and &mut AltMatch arguments
supplied to a callback, and JavaScript function references are registered as
typed Rust callbacks before loading serialized JSON. Unsupported or mistyped
serialized callback forms fail at install time instead of being silently
ignored.
§Gaps against the canonical surface
Audited against ts/src (tabnas.ts, parser.ts, lexer.ts, rules.ts,
context.ts, utility.ts) in September 2026. Parse behavior is not on this
list: where a behavior differs it is a defect, it is repaired, and until it is
repaired it is registered in test/spec/divergent.tsv, whose rust column
this crate’s own suite asserts. One entry there is Rust’s alone today, the
chain walk named above.
What follows is public API surface that TypeScript offers a plugin author and
this crate does not, or offers in another shape.
- The shared utility bag. TypeScript exports a
utilobject and this crate’stabnas::utilitycarries four of its members:deep,modlist,str_valueandstr_inject, which are the ones the sharedutility-*.tsvfixtures pin.escre,mesc,regexp,getpath,charset,charsBitmap,snip,srcfmt,clean,configure,filterRules,makelog,badlex,findTokenSet,keyOrder,recordKeyOrderandKEY_ORDERhave no public Rust equivalent. Several exist inside the crate and are simply not exported; the rest are internal to how the TypeScript engine is built. The members that only shim JavaScript (isarr,omap,keys,values,entries,assign,clone,str) have no meaning here and are not gaps. - The matcher factories. TypeScript exports
makeCommentMatcher,makeFixedMatcher,makeLineMatcher,makeNumberMatcher,makeSpaceMatcher,makeStringMatcher,makeTextMatcherandmakeLex, so a plugin can build a matcher from config and install it beside the built-in bands. Here the eight bands are internal and a plugin reaches the same outcome throughlex_match_ref,imperative_lex_match_refandlex_match_factory_ref, which compose rather than construct. The capability is present; the shape is not the same. - Constructors exported for plugins.
makeRule,makeRuleSpec,makeToken,makePointandmakeParserare public in TypeScript. HereRule,TokenandParserare constructed by the engine, and a plugin receives them. - Per-alternate validation of a grammar held as data. TypeScript exports
validateAltandvalidateAltsalongsidevalidateGrammar, and Go exportsValidateAlt/ValidateAlts, so a generator or an editor can check one alternate before any parser exists. This crate exposesgrammar::validate_grammarover a whole rule table and nothing finer. - The builtin reference set is not public. TypeScript exports
BUILTIN_REFSandBUILTIN_SCHEMA_VERSION, and Go exports both under the same names, so a compiler emitting pure-data grammars can ask the engine which builtin references it honours. HereBUILTIN_SCHEMA_VERSIONis public astabnas::grammar::BUILTIN_SCHEMA_VERSIONand is not re-exported at the crate root, and the name set ispub(crate). The three sets are not identical between the runtimes, so exposing one here is a contract decision rather than a transcription, and it is left to the maintainer. internal(). TypeScript hands back one bag of engine internals. The same material is reachable here through named accessors (config,rule_names,rule_specs,token_set,token_name,installed_plugins,describe), which is a deliberate difference and not an absence.
§Building a release that uses this
The engine is a separate crate from whatever links it, so nothing in it can be inlined into your code unless your binary asks for that. Put this in the release profile of the crate that builds the final artefact:
[profile.release]
opt-level = 3
codegen-units = 1
lto = "fat"lto is the one that matters and the one that is not on by default.
Without it, tabnas/measure records this port as 1.10x to 1.15x
slower on every case. Go’s compiler inlines across packages within a
binary by default and V8 inlines across module boundaries at runtime, so
this is the flag that puts a Rust consumer on the same footing rather
than an unusual tuning.
§Pick an allocator
The other two ports of this engine bring their own: Go has the
runtime’s and TypeScript has V8’s. A Rust binary takes whatever libc
supplies unless it says otherwise, and this engine allocates heavily
enough for that to matter. On glibc, tabnas/measure records
1.09x to 1.56x between the system allocator and mimalloc, the
larger end on the deeply nesting grammars.
[dependencies]
mimalloc = "0.1"#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;mimalloc is what the measurement harness uses; jemalloc and others
are reasonable too. The point is the choice, not the crate: leaving it
to the platform is itself a choice, and on glibc it is an expensive
one.
A library cannot set a profile or an allocator for its consumers, which is why both of these are written down rather than configured.
§Development
cargo build --all-targets
cargo test --all-targets
cargo test --doc
cargo clippy --all-targets --all-features -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --no-depsThe crate declares Rust 1.85 as its minimum supported toolchain. From the
repository root, ./ci/rust/run.sh also runs the shared cross-runtime and
compiler-consumer parity gates.
§Formal verification (experiment, not a gate)
rs/verus/ holds standalone Verus
copies of two pieces of src/, with specifications and machine-checked
proofs: the InlineText length invariant behind the crate’s only
unsafe block, and the lexer’s scalar-index span arithmetic.
Nothing there is compiled by the crate. Verus pins its own Rust
toolchain, which is not the 1.85 above, so no gate runs it and it is not
a build dependency. Run it by hand with VERUS=/path/to/verus rs/verus/run.sh. For the unsafe block, the proof covers the length
bound that keeps the slice in range, and not the UTF-8 validity it also
relies on, because Verus has no specification for
str::from_utf8_unchecked. The crate itself is not written for Verus:
wrapping code in verus!{} spreads to everything that code touches, and
the engine’s Rc, RefCell, dyn and HashMap sites are outside the
subset Verus verifies.
Re-exports§
pub use context::ActionError;pub use context::Context;pub use context::ContextSeed;pub use context::InstanceInfo;pub use error::RecoveredAt;pub use error::TabnasError;pub use grammar::GrammarError;pub use grammar::GrammarGroups;pub use grammar::GrammarSetting;pub use grammar::GrammarSettingAlt;pub use grammar::GrammarSettingRule;pub use grammar::GrammarSpec;pub use lexer::Lexer;pub use lexer::RelexCheckpoint;pub use options::MAX_RULE_HISTORY;pub use options::BudgetCheck;pub use options::BudgetOptions;pub use options::ColorOptions;pub use options::CommentDef;pub use options::CommentSuffixMatcher;pub use options::ConfigModifier;pub use options::ContextParsePrepare;pub use options::DebugOptions;pub use options::DebugOutput;pub use options::DebugPrintOptions;pub use options::DebugSourceFormatter;pub use options::ErrMsgOptions;pub use options::ErrorSuffix;pub use options::ErrorSuffixCallback;pub use options::ErrorSuffixContext;pub use options::FixedOptions;pub use options::FixedToken;pub use options::ImperativeCommentSuffixMatcher;pub use options::ImperativeLexCheck;pub use options::ImperativeLexMatcher;pub use options::ImperativeTextModifier;pub use options::InfoOptions;pub use options::LexCheck;pub use options::LexCheckResult;pub use options::LexCheckToken;pub use options::LexMatcher;pub use options::LexMatcherCallback;pub use options::LexMatcherFactory;pub use options::ListOptions;pub use options::MapMerge;pub use options::MapOptions;pub use options::MatchToken;pub use options::MatchTokenCallback;pub use options::MatchTokenMatcher;pub use options::MatchTokenResult;pub use options::MatchValue;pub use options::Options;pub use options::ParseOptions;pub use options::ParsePrepare;pub use options::ParsePrepareWithInstance;pub use options::ParserOptions;pub use options::ParserStart;pub use options::ParserStartWithContext;pub use options::ParserStartWithInstance;pub use options::RecoverOptions;pub use options::ResultOptions;pub use options::RewindOptions;pub use options::SafeOptions;pub use options::SpaceOptions;pub use options::TextModifier;pub use options::ValueDef;pub use options::ValueOptions;pub use options::ValueTextModifier;pub use options::ValueTransform;pub use parser::Continuations;pub use parser::ParseRecovery;pub use parser::Parser;pub use rule::ActionBinding;pub use rule::AltAction;pub use rule::AltActionBinding;pub use rule::AltBack;pub use rule::AltBackWithMatch;pub use rule::AltCondition;pub use rule::AltConditionWithLexer;pub use rule::AltConditionWithLexerAndMatch;pub use rule::AltConditionWithMatch;pub use rule::AltError;pub use rule::AltErrorWithMatch;pub use rule::AltMatch;pub use rule::AltModifier;pub use rule::AltModifierWithMatch;pub use rule::AltNext;pub use rule::AltNextWithMatch;pub use rule::AltSpec;pub use rule::CompareOp;pub use rule::Condition;pub use rule::Rule;pub use rule::RuleDone;pub use rule::RuleDoneAlt;pub use rule::RuleName;pub use rule::RuleSnapshot;pub use rule::RuleSpec;pub use rule::RuleState;pub use rule::StateAction;pub use token::name_to_tin;pub use token::tin_name;pub use token::Point;pub use token::Site;pub use token::Tin;pub use token::Token;pub use token::TokenCode;pub use token::TokenText;pub use token::TokenValFunc;pub use token::TIN_AA;pub use token::TIN_BD;pub use token::TIN_CA;pub use token::TIN_CB;pub use token::TIN_CL;pub use token::TIN_CM;pub use token::TIN_CS;pub use token::TIN_LN;pub use token::TIN_MAX;pub use token::TIN_NR;pub use token::TIN_OB;pub use token::TIN_OS;pub use token::TIN_SP;pub use token::TIN_ST;pub use token::TIN_TX;pub use token::TIN_UK;pub use token::TIN_VL;pub use token::TIN_ZZ;pub use value::ListRef;pub use value::MapRef;pub use value::Text;pub use value::Value;
Modules§
- builtins
- context
- error
- grammar
- Loading and installing the JSON-serializable grammar interchange format.
- lexer
- options
- parser
- rule
- token
- utility
- Cross-runtime utility primitives used by grammar and option handling.
- value
Structs§
- Decoration
- A type-erased value attached to a parser instance by a native plugin.
- Merge
Error - Plugin
- Native Rust plugin descriptor. The explicit name replaces JavaScript’s
Function.nameand provides a stable namespace for plugin options. - Plugin
Error - Tabnas
Constants§
Type Aliases§
- Action
- Context
Action - LexSubscriber
- Parse
Guard - A check the parse loop runs at every step; see
Tabnas::parse_guard. - Plugin
Callback - Rule
Done Subscriber - Rule
Subscriber - Token
Subscriber