tokora 0.10.0

Deterministic parser combinators, with on-demand lexing, LALR-style dispatch, explicit backtracking, and configurable diagnostics.

Introduction

Tokora is a Rust parser-combinator library with on-demand lexing, explicit lookahead and backtracking, configurable diagnostics, and optional Logos and Rowan integrations. Parsers work over a Lexer and Token model, so the same grammar can use a fail-fast runtime emitter or a collecting diagnostic emitter.

Install

Most applications use the maintained Logos adapter:

[dependencies]
tokora = { version = "0.10", features = ["logos"] }

logos is the alias for the logos_0_16 integration, the only Logos major tokora supports. The default std feature remains enabled unless you set default-features = false.

Which version this describes. Every [dependencies] snippet in this README resolves against the latest release, 0.10.0. A feature that main has grown since is named with the version it will arrive in, rather than put in a copyable block you cannot resolve.

Capabilities

• On-demand token flow through InputRef, with explicit cache-backed lookahead and transactions.
• Plain parser functions plus composable sequencing, repetition, delimiters, and deterministic choice.
• Token-level and AST-level Pratt parsing.
• Configurable Fatal, Verbose, Silent, and Ignored diagnostics.
• Recovery, partial-input support, lexer conformance checks, tracing, and a public fuzz harness.
• Optional lossless CST building over the parser's own backtracking (feature rowan): a rewindable event-stream sink where a parser rollback rewinds the half-built tree, and node combinators bracket sub-parses into syntax nodes.
• Optional adapters for Logos, Rowan CSTs, source types, and container types.

How Tokora parses

A Tokora grammar is ordinary Rust: parser functions and combinators read through InputRef, which pulls tokens from a Lexer on demand and stages tokens in its cache when lookahead or backtracking needs them. peek_then_choice makes a decision from a fixed lookahead window; dispatch_on_kind and fused_dispatch_on_kind route the next token's Token::Kind to exactly one selected branch.

That token-kind dispatch is local to a hand-written combinator grammar. Tokora does not accept an LALR grammar, generate LALR parse tables, or act as an LALR parser generator.

When a grammar needs speculation, it is explicit. attempt and try_attempt commit successful work and roll back a decline or error; Transaction exposes commit and rollback directly. A rollback restores the input position, span, lexer state, token cache, and diagnostics emitted since the checkpoint. Application-owned side effects need their own transaction boundary.

Diagnostics and recovery

Parsers are generic over their parse context, including the emitter. Parser::new() uses the fail-fast Fatal emitter; Verbose records diagnostics and can continue when the grammar recovers. The same parser functions can therefore serve a runtime parser, compiler front end, or editor integration without a second grammar implementation.

Structured lexer, token, separator, container, and Pratt errors convert into the application's error type through From implementations.

Recovery is explicit: recover restores the failed parse's starting point before running a recovery parser, while inplace_recover continues from the failure position. sync_balanced and skip_then_retry provide nesting-aware synchronization; Verbose records each successful non-empty skipped region once alongside other diagnostics. Incomplete errors are re-raised instead of recovered so unfinished partial input is not discarded.

Recursion budget

Each InputRef::descend a grammar takes, and each frame of tokora's own Pratt engines, draws on one shared cell, so an input nested past the ceiling fails the parse with a catchable tokora::error::RecursionLimitReached instead of exhausting the native stack. The default is RecursionLimiter::PARSE_DEFAULT_DEPTH, now public rather than pub(crate), and 0.10.0 lowers it from 64 to 32 — a parse that relied on the 0.9.1 ceiling without configuring one has to ask for the depth it needs.

32 is also the number in a release build, and that is a decision rather than a missing measurement. A release frame is roughly an order of magnitude cheaper than a debug one, and the release rows do support 256; what is missing is any way for this crate to know that the condition holds. debug_assertions is not opt-level — a build with debug-assertions = false at opt-level = 0 selects the release arm while paying unoptimised frame prices — and it is per crate, while the frames this budget bounds are the caller's own productions. tokora shipped that divergence for one revision and it was a process-level abort with nothing on any Result channel, so both arms are now priced at the debug frame cost and no profile combination can abort.

The release figure is published rather than installed, and taking it is one call:

use tokora::state::recursion_tracker::RecursionLimiter;

// The default budget — the same number in every build profile.
assert_eq!(RecursionLimiter::PARSE_DEFAULT_DEPTH, 32);

// The depth a fully optimised parse supports. Nothing installs it; a caller does.
assert_eq!(RecursionLimiter::OPTIMIZED_PARSE_DEPTH, 256);

let limiter = RecursionLimiter::with_limitation(RecursionLimiter::OPTIMIZED_PARSE_DEPTH);
assert_eq!(limiter.limitation(), 256);

Hand that limiter to ParserContext::with_recursion_limiter, or to InputContext::with_recursion_limiter for a driver that builds its own input context — no new API is involved. Pass 256 when every frame the budget bounds is compiled at opt-level = 3, tokora's and your own productions; a per-package profile override, or a debug consumer against a release tokora, puts the default back in force. RecursionLimiter::SEGMENTED_PRATT_DEPTH is a third published figure — 1024, stacker-only — for a descent that is entirely Pratt frames, and What stacker segments is why it needs its own constant.

Guide and examples

The Tokora Guide is five parts and 28 chapters. Part II is the tutorial — ten chapters that build Calc end to end, from tokens and the lexer through composition, deterministic choice, Pratt expressions, backtracking, diagnostics, recovery, partial input, and testing. Part III is the internals, for a reader who wants to know why an API is shaped the way it is: the parse-while-lexing engine, checkpoint/rewind and the LIFO contract, the atomic emitter, the event-stream CST engine, and Source/Slice storage backends. Part IV applies it — an anatomy chapter, a custom-lexer recipe, the four maintained-example walkthroughs, and the Rowan lossless-CST chapter. Part V is reference: combinators and atoms, errors, emitters and context, vocabulary, macros and feature flags, Pratt, and the types and syntax building blocks.

The examples below are canonical complete programs; the guide links back to them instead of copying whole files into prose.

The book source lives under tokora/src/guide, and the examples also compile together with cargo test -p tokora --no-default-features --features std,logos,combinators --examples.

Features

The combinator-family gates — combinators and the thirteen families it covers, any through validate — are new in 0.9.0. On 0.8.0 and earlier there are no per-family gates, default is std alone, and every combinator is compiled unconditionally.

Every combinator family is independently gateable so an embedded or no-alloc build compiles only the combinators it calls. What the families sit on stays unconditional: Parser/Parse/parse*, the ParseInput / TryParseInput / ParseChoice traits, and the substrate combinators (expect, delimited, recover, select, opt, padded, node, labelled, …). combinators is a default feature, so a plain dependency line sees the whole surface; a default-features = false build names the families it uses.

Feature aliases select their versioned counterpart; versioned features make the corresponding optional dependency available. tokora::logos and the unversioned tokora::lexer::LogosLexer are available with logos_0_16 and re-export/adapt that version — the only Logos major tokora supports. rowan does not enable logos, and smallvec_1 is the versioned feature that adds alloc.

What stacker segments

stacker puts a fresh heap stack segment under the frame prologue of tokora's two Pratt engines, and under nothing else. A consumer's own descend/descending frames are ordinary native frames and are untouched, so the feature does not move RecursionLimiter::PARSE_DEFAULT_DEPTH, the budget they share. RecursionLimiter::SEGMENTED_PRATT_DEPTH is the larger figure it does justify, for a caller whose whole descent is Pratt frames to opt into.

It is not a substitute for the recursion budget: a segment is an mmap, so a deep enough input still ends the process with nothing on any Result channel.

rowan and known upstream undefined behaviour

A lossless sink requires a trivia-surfacing lexer (Lexer::SURFACES_TRIVIA). Add rowan = "0.17" directly when implementing rowan::Language.

Safety disclosure: rowan 0.17.0 executes known undefined behaviour on the ordinary construct-and-drop path — Stacked Borrows at arc.rs:264, reached from building any green tree, and Tree Borrows at cursor.rs:136, reached from dropping any red-tree SyntaxNode. tokora's own src/cst contains no unsafe; the defects are upstream and unfixed since 2021 (rust-analyzer/rowan #108, #163, #192), so raising the requirement is not an exit. Both Miri matrices exclude this feature for that reason, which means the shipped lossless path has zero Miri coverage rather than a green result. Tracked at al8n/tokora#252.

Platform support

Tokora's MSRV is Rust 1.95. Tokora's core supports both allocator-free no_std (no_std without alloc) and allocation-enabled no_std (no_std with alloc). Disable default features for allocator-free core use. Enable alloc when a parser, cache, or selected optional facility requires allocation; other optional facilities may require std.

Allocator-free no_std:

[dependencies]
tokora = { version = "0.10", default-features = false }

no_std with alloc:

[dependencies]
tokora = { version = "0.10", default-features = false, features = ["alloc"] }

Neither line enables a combinator family: combinators is a default feature, and both turn the defaults off. Add features = ["combinators"] for the umbrella, or list the families the grammar actually calls, as in features = ["alloc", "many", "map"].

Design philosophy and inspirations

Core Priorities

1. Performance - Pull tokens from the lexer on demand and offer fused dispatch where avoiding a peek/cache round trip matters.
2. Predictability - Prefer deterministic lookahead and token-kind dispatch; make speculation explicit and transactional.
3. Composability - Combine small parser functions and combinators; compose focused emitter traits into custom diagnostic strategies.
4. Versatility - Reuse parser functions with fail-fast, collecting, silent, or custom emitters.
5. Flexibility - Work through generic Lexer and Token traits, with optional Logos input and Rowan CST integrations.
6. Correctness - Track spans and structured errors, rewind emitted diagnostics with parser rollbacks, and provide conformance and fuzz test kits.

Inspirations

Tokora takes inspiration from:

• winnow - For ergonomic parser API design
• chumsky - For composable parser combinator patterns
• logos - For high-performance lexing
• rowan - For lossless syntax tree representation

Development

Useful repository checks:

cargo fmt --all --check
cargo test -p tokora --all-features
cargo test -p tokora --no-default-features --features std,logos,combinators --examples
RUSTDOCFLAGS="-D warnings" cargo test -p tokora --all-features --doc
python3 tokora/tools/validate_docs.py --source
(cd tokora && mdbook build)
python3 tokora/tools/validate_docs.py --book target/book

The guide is validated both as rustdoc and as an mdBook so API links, local links, chapter order, and Pages output stay aligned.

License

tokora is under the terms of both the MIT license and the Apache License (Version 2.0).

See the Apache License, Version 2.0 and the MIT license text for details.

Copyright (c) 2026 Al Liu.