Part of Ferramenta, a family of Rust tools.

Ferroni — Oniguruma, continued in Rust

The regex engine behind TextMate grammars, jq and PHP's mbregex, continued in memory-safe Rust, with the vscode-oniguruma scanner built in.

Oniguruma sits underneath a large part of the text tooling world. TextMate grammars, and with them VS Code, Shiki and most syntax highlighters, are written in its syntax. jq and PHP's mbregex use it directly, and Ruby's engine, Onigmo, started as a fork of it. Its feature set goes well beyond most regex libraries: variable-length look-behind, conditionals, absent expressions, subexpression calls, 902 Unicode properties, and 12 syntax modes from Perl to POSIX.

The C project ended on April 24, 2025. Ferroni carries the engine forward in Rust:

Same engine, verified. A line-by-line port that keeps Oniguruma's module structure and optimization pipeline, not a lookalike. Every upstream UTF-8 test passes (test parity).
Memory-safe, no C toolchain. cargo add ferroni and build: no bindgen, no C compiler. C Oniguruma has a long history of memory-safety CVEs; Ferroni keeps unsafe at 0.4%, every block documented in ADR-002.
Faster where highlighters spend their time. Tokenizing real code with complete TextMate grammars is between 2x and 30x faster than the C original, and text search is up to 6x faster (Performance).
The vscode-oniguruma scanner, built in. vscode-textmate and Shiki tokenize through vscode-oniguruma's multi-pattern scanner. Ferroni ships a Scanner API of the same shape, UTF-16 offsets included, in the same crate.

Why Ferroni exists

We build Ferromark, a Markdown engine in Rust, and Ardo, a documentation framework, and we want the whole chain in Rust, including syntax highlighting on par with Shiki. That highlighter is Ferriki, a Shiki-compatible engine that tokenizes TextMate grammars with Ferroni's scanner. Those grammars are written for Oniguruma: they rely on look-behind, \G anchors, backreferences and other features most regex engines skip.

Bindings to the C library would have kept the C code, the C toolchain and its memory-safety record. A new engine would have broken the grammars in subtle ways. So we ported Oniguruma itself, line by line, verified the port against its own test suite, and then tuned the path that highlighters actually take.

Ferroni is not a replacement for Rust's regex crate. When a pattern fits regex's narrower syntax, regex is usually faster. Ferroni is for workloads that need Oniguruma's features and behavior.

Quick start

Add to your Cargo.toml:

[dependencies]
ferroni = "1"

MSRV: Rust 1.94 (enforced in CI).

Regex

use ferroni::prelude::*;

fn main() -> Result<(), RegexError> {
    let re = Regex::new(r"(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})")?;

    let caps = re.captures("Date: 2026-02-12").unwrap();
    assert_eq!(caps.get(0).unwrap().as_str(), "2026-02-12");
    assert_eq!(caps.name("year").unwrap().as_str(), "2026");
    assert_eq!(caps.name("month").unwrap().as_str(), "02");
    Ok(())
}

Scanner API

The Scanner matches multiple patterns simultaneously -- the core operation behind TextMate-based syntax highlighting. Results include UTF-16 position mapping for direct use with vscode-textmate and Shiki.

use ferroni::scanner::{Scanner, ScannerFindOptions};

let mut scanner = Scanner::new(&[
    r"\b(function|const|let|var)\b",  // keywords
    r#""[^"]*""#,                      // strings
    r"//.*$",                          // comments
]).unwrap();

let code = r#"const x = "hello" // greeting"#;
let m = scanner.find_next_match(code, 0, ScannerFindOptions::NONE).unwrap();

assert_eq!(m.index, 0); // pattern 0 matched first ("const")
assert_eq!(m.capture_indices[0].start, 0);
assert_eq!(m.capture_indices[0].end, 5);

For fine-grained control, use RegexBuilder:

use ferroni::prelude::*;

let re = Regex::builder(r"hello")
    .case_insensitive(true)
    .build()
    .unwrap();
assert!(re.is_match("Hello World"));

The full C-ported API is also available for advanced usage:

use ferroni::regcomp::onig_new;
use ferroni::regexec::onig_search;
use ferroni::oniguruma::*;
use ferroni::regsyntax::OnigSyntaxOniguruma;

let reg = onig_new(
    b"\\d{4}-\\d{2}-\\d{2}",
    ONIG_OPTION_NONE,
    &ferroni::encodings::utf8::ONIG_ENCODING_UTF8,
    &OnigSyntaxOniguruma,
).unwrap();

let input = b"Date: 2026-02-12";
let (result, region) = onig_search(
    &reg, input, input.len(), 0, input.len(),
    Some(OnigRegion::new()), ONIG_OPTION_NONE,
);

assert!(result >= 0);
assert_eq!(result, 6); // match starts at byte 6

Both APIs have a runnable example in examples/:

cargo run --example basic_match       # matching, captures, iteration
cargo run --example scanner_tokenize  # multi-pattern scanner, UTF-16 offsets

Supported features

Scanner -- multi-pattern matching with result caching, two search strategies (RegSet for short strings, per-regex for long strings), and automatic UTF-16 position mapping. API-compatible with vscode-oniguruma.

Full Oniguruma regex -- every feature from the C engine:

All Perl/Ruby/Python syntax -- (?:...), (?=...), (?!...), (?<=...), (?<!...), (?>...)
Named captures -- (?<name>...), (?'name'...), (?P<name>...)
Backreferences -- \k<name>, \g<name>, relative \g<-1>
Conditionals -- (?(cond)T|F)
Absent expressions -- (?~...)
Unicode properties -- \p{Greek}, \p{Lu}, \p{Emoji} (902 names) from Unicode 17.0 data; C Oniguruma's last snapshot has Unicode 16.0 (ADR-015)
Grapheme clusters -- \X, text segment boundaries \y, \Y
Callouts -- (?{...}), (*FAIL), (*MAX{n}), (*COUNT), (*CMP)
12 syntax modes -- Oniguruma, Ruby, Perl, Perl_NG, Python, Java, Emacs, Grep, GNU, POSIX Basic/Extended, ASIS
Safety limits -- retry, time, stack, subexp call depth (global + per-search)

Performance

This summary draws on a recorded battle_bench run. Scanner comparisons use complete, unmodified Shiki TextMate grammars for TypeScript, CSS, and Rust. It shows the direction across the measured workloads, not a promise for every pattern or machine. Raw results, inputs, and measurement context are in Benchmark Results.

Scanner workloads

For syntax highlighting, Ferroni consistently spends less time finding the next match and tokenizing lines than Oniguruma, across all three production grammars. How large the lead is depends on the workload. Tokenizing whole documents line by line, each line handed to the scanner once as a real tokenizer does, Ferroni is more than twice as fast on TypeScript, about 10x faster on Rust, and about 30x faster on CSS (exact numbers). Re-scanning the same line, where Ferroni can reuse what it learned about that string, the lead grows to between about 15x and nearly 300x (exact numbers). Compiling the full TypeScript and CSS grammars is also faster in Ferroni, while Oniguruma still starts faster on the smaller Rust grammar. In practice, Ferroni's strongest fit is a scanner that compiles a grammar once and uses it repeatedly.

Search and matching

On the measured log-style searches, Ferroni is faster than Oniguruma for both matches and misses, capture-based extraction, timestamps, and multi-pattern search. The advantage ranges from modest to several-fold, spread across cases rather than resting on one standout case. Ferroni also comes out ahead across the measured Oniguruma-compatible matching patterns, including lookaround, Unicode properties, backreferences, and alternation.

When patterns fit the narrower syntax of Rust's regex crate, that engine often has the throughput advantage. It is a good choice for plain-text matching when its feature set is enough; Ferroni is for workloads that need Oniguruma features and practical scanner speed.

Compilation and memory

Compile-time results are mixed. Ferroni compiles the full TypeScript and CSS grammars faster, but Oniguruma has an edge on Rust grammar startup and on some individual patterns. The regex crate can take longer to compile, especially for capture-heavy patterns. In a separate process-level memory measurement, Ferroni and Oniguruma peak in the same approximate range while compiling and scanning a large TypeScript workload. Details and methodology are in the Memory Measurements.

Ferroni keeps benchmark suites separated by purpose:

Internal suite (regression_bench, Rust-only, regression/optimization work):

cargo bench
cargo bench --bench regression_bench

# compare two local baselines
cargo bench --bench regression_bench -- --save-baseline main
# switch to the branch you want to compare
cargo bench --bench regression_bench -- --baseline main

Reference suite (battle_bench, Ferroni vs Oniguruma for publishable numbers):
```
# one-time setup
./scripts/prepare-oniguruma-sources.sh

cargo bench --features ffi --bench battle_bench
```
Exact external input pins for this suite live in benches/battle_inputs.toml.
Memory comparison (process-isolated peak RSS on the large TypeScript scanner workload):
```
./scripts/run-battle-memory.sh
```

HTML report:

open target/criterion/report/index.html

Architecture

Each C source file maps 1:1 to a Rust module (ADR-001):

C File	Rust Module	Purpose
regparse.c	`regparse.rs`	Pattern parser
regcomp.c	`regcomp.rs`	AST-to-bytecode compiler
regexec.c	`regexec.rs`	VM executor
regint.h	`regint.rs`	Internal types and opcodes
oniguruma.h	`oniguruma.rs`	Public types and constants
regenc.c	`regenc.rs`	Encoding trait
regsyntax.c	`regsyntax.rs`	12 syntax definitions
regset.c	`regset.rs`	Multi-regex search (RegSet)
regerror.c	`regerror.rs`	Error messages
regtrav.c	`regtrav.rs`	Capture tree traversal
unicode.c	`unicode/mod.rs`	Unicode tables and segmentation
--	`scanner.rs`	Multi-pattern scanner for syntax highlighting

Compilation pipeline (same as C):

onig_new() -> onig_compile()
  -> onig_parse_tree()     (pattern -> AST)
  -> reduce_string_list()  (merge adjacent strings)
  -> tune_tree()           (6 optimization sub-passes)
  -> compile_tree()        (AST -> VM bytecode)
  -> set_optimize_info()   (extract search strategy)

Scope

Ferroni targets ASCII/UTF-8 workloads. The following are intentionally not included:

27 of 29 encodings -- only ASCII and UTF-8 (ADR-003). UTF-16/32, Shift_JIS, EUC-JP/KR/TW/CN, Big5, GB18030, KOI8-R, CP1251, and ISO-8859-1 to -16 are not ported. TextMate scanning (vscode-oniguruma compiles every pattern as UTF-8) and jq are fully covered; PHP's mbregex in a non-UTF-8 encoding is not.
POSIX/GNU API -- regcomp/regexec/regfree (ADR-012)
C memory management -- replaced by Rust's Drop trait
onig_new_deluxe -- C-specific allocation, use onig_new() instead

Running tests

The test suites, their stack requirements, and the coverage gate are described in CONTRIBUTING.md.

Test parity

Every upstream test that targets UTF-8 is ported 1:1. EUC-JP tests are out of scope (ADR-003).

Upstream file	Encoding	Upstream	Ferroni	Status
test_utf8.c	UTF-8	1,554	1,561	100%
test_back.c	UTF-8	1,225	1,225	100%
test_syntax.c	UTF-8	144	144	100%
test_options.c	UTF-8	47	48	100%
test_regset.c	UTF-8	4	15	100%
testc.c	EUC-JP	785	—	out of scope
C total (UTF-8)		2,974	2,993	100%
vscode-oniguruma tests	UTF-16	15	25	100%

On top of the ported upstream cases, Ferroni adds Rust-native tests for API integration, edge cases, error paths, and coverage gaps. The tree holds 2,349 #[test] functions in total (some compat functions bundle multiple upstream cases, so this is higher than the parity totals above). The count is derived from the tree by scripts/count-tests.sh, which is the single source for the test counts quoted in this README and in CONTRIBUTING.

C Oniguruma has no coverage reporting. Ferroni's test suite is a strict superset of the upstream tests.

Line coverage: 87.7%, gated at 87%. Measured with cargo-llvm-cov by scripts/coverage.sh, which CI runs on every pull request and which fails the build below the gate -- the threshold is defined in that script. Generated Unicode tables, the FFI bindings and the test files themselves are excluded from the measurement. 42 deeply recursive tests are skipped under LLVM instrumentation (stack overflow from coverage bookkeeping) but pass in normal cargo test.

Architecture decision records

ADR	Decision
001	1:1 structural parity with C original
002	Unsafe code policy
003	Encoding scope: ASCII and UTF-8 only
004	C-to-Rust translation patterns
005	Idiomatic Rust API layer
006	Scanner API for TextMate tokenization
007	SIMD-accelerated search via memchr
008	Rust-only optimizations and performance philosophy
009	Dependency philosophy
010	Benchmark strategy
011	Test strategy and C test suite parity
012	POSIX and GNU API not ported
013	Stack overflow mitigation in debug builds
014	Porting bugs: lessons learned
015	Generate Unicode tables from the UCD

Contributing

Contributions are welcome! Please read CONTRIBUTING.md and review the ADRs before submitting a PR.

Acknowledgments

Ferroni is built on the work of K. Kosako and the Oniguruma contributors. The C original powers regex in Ruby, PHP, TextMate, jq, and many other projects. The Scanner API and its test suite are based on vscode-oniguruma by Microsoft and the VS Code team.

Oniguruma ended in April 2025. Ferroni is an independent continuation by Sebastian Software; it is not affiliated with or endorsed by the original author.

License

BSD-2-Clause (same as Oniguruma)

Ferroni is the one Ferramenta tool that is not published under MIT OR Apache-2.0: it is a line-by-line port of Oniguruma's C source, so it stays under the BSD-2-Clause license of the code it is derived from.

More from Ferramenta

Ferramenta — A family of Rust tools.

The content pipeline

Tool	Job
ferriki	Shiki-compatible syntax highlighting
ferromark	Markdown to HTML, sanitized by default

The language workshop

Tool	Job
ferrolex	Spell checking for text and code
ferrocat	Translation catalog engine
palamedes	Internationalization for TypeScript applications

On the workbench

Tool	Job
ferralk	Glob matching and parallel filesystem walking
ferrugo	PDF previews for untrusted files
dalo	Team agent skills, versioned and synced as code
cuttledoc	Local-first speech transcription

ferroni 1.7.0