Part of Ferramenta, a family of Rust tools.

Ferroni — Oniguruma, continued in Rust

The regex engine behind TextMate grammars, jq and PHP's mbregex, continued in memory-safe Rust, with the vscode-oniguruma scanner built in.

Oniguruma sits underneath a large part of the text tooling world. TextMate grammars, and with them VS Code, Shiki and most syntax highlighters, are written in its syntax. jq and PHP's mbregex use it directly, and Ruby's engine, Onigmo, started as a fork of it. Its feature set goes well beyond most regex libraries: variable-length look-behind, conditionals, absent expressions, subexpression calls, 902 Unicode properties, and 12 syntax modes from Perl to POSIX.

The C project ended on April 24, 2025. Ferroni carries the engine forward in Rust:

Same engine, verified. A line-by-line port that keeps Oniguruma's module structure and optimization pipeline, not a lookalike. Every upstream UTF-8 test passes (test parity).
Memory-safe, no C toolchain. cargo add ferroni and build: no bindgen, no C compiler. C Oniguruma has a long history of memory-safety CVEs; Ferroni keeps unsafe at 0.4%, every block documented in ADR-002.
Faster where highlighters spend their time. Tokenizing real code with complete TextMate grammars is between 2x and 30x faster than the C original, and text search is up to 6x faster (Performance).
The vscode-oniguruma scanner, built in. vscode-textmate and Shiki tokenize through vscode-oniguruma's multi-pattern scanner. Ferroni ships a Scanner API of the same shape, UTF-16 offsets included, in the same crate.

Why Ferroni exists

We build Ferromark, a Markdown engine in Rust, and Ardo, a documentation framework, and we want the whole chain in Rust, including syntax highlighting on par with Shiki. That highlighter is Ferriki, a Shiki-compatible engine that tokenizes TextMate grammars with Ferroni's scanner. Those grammars are written for Oniguruma: they rely on look-behind, \G anchors, backreferences and other features most regex engines skip.

Bindings to the C library would have kept the C code, the C toolchain and its memory-safety record. A new engine would have broken the grammars in subtle ways. So we ported Oniguruma itself, line by line, verified the port against its own test suite, and then tuned the path that highlighters actually take.

Ferroni is not a replacement for Rust's regex crate. When a pattern fits regex's narrower syntax, regex is usually faster. Ferroni is for workloads that need Oniguruma's features and behavior.

Quick start

Add to your Cargo.toml:

[dependencies]
ferroni = "1"

MSRV: Rust 1.94 (enforced in CI).

Regex

use ferroni::prelude::*;

fn main() -> Result<(), RegexError> {
    let re = Regex::new(r"(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})")?;

    let caps = re.captures("Date: 2026-02-12").unwrap();
    assert_eq!(caps.get(0).unwrap().as_str(), "2026-02-12");
    assert_eq!(caps.name("year").unwrap().as_str(), "2026");
    assert_eq!(caps.name("month").unwrap().as_str(), "02");
    Ok(())
}

Scanner API

The Scanner matches multiple patterns simultaneously -- the core operation behind TextMate-based syntax highlighting. Results include UTF-16 position mapping for direct use with vscode-textmate and Shiki.

use ferroni::scanner::{Scanner, ScannerFindOptions};

let mut scanner = Scanner::new(&[
    r"\b(function|const|let|var)\b",  // keywords
    r#""[^"]*""#,                      // strings
    r"//.*$",                          // comments
]).unwrap();

let code = r#"const x = "hello" // greeting"#;
let m = scanner.find_next_match(code, 0, ScannerFindOptions::NONE).unwrap();

assert_eq!(m.index, 0); // pattern 0 matched first ("const")
assert_eq!(m.capture_indices[0].start, 0);
assert_eq!(m.capture_indices[0].end, 5);

A grammar builds one scanner per rule context, and the same patterns recur across them. Build them with Scanner::with_pattern_cache and one ScannerPatternCache, and each distinct pattern is compiled once and shared; the cache is yours to keep or drop, with no process-wide state (ADR-006).

For fine-grained control, use RegexBuilder:

use ferroni::prelude::*;

let re = Regex::builder(r"hello")
    .case_insensitive(true)
    .build()
    .unwrap();
assert!(re.is_match("Hello World"));

The full C-ported API is also available for advanced usage:

use ferroni::regcomp::onig_new;
use ferroni::regexec::onig_search;
use ferroni::oniguruma::*;
use ferroni::regsyntax::OnigSyntaxOniguruma;

let reg = onig_new(
    b"\\d{4}-\\d{2}-\\d{2}",
    ONIG_OPTION_NONE,
    &ferroni::encodings::utf8::ONIG_ENCODING_UTF8,
    &OnigSyntaxOniguruma,
).unwrap();

let input = b"Date: 2026-02-12";
let (result, region) = onig_search(
    &reg, input, input.len(), 0, input.len(),
    Some(OnigRegion::new()), ONIG_OPTION_NONE,
);

assert!(result >= 0);
assert_eq!(result, 6); // match starts at byte 6

Both APIs have a runnable example in examples/:

cargo run --example basic_match       # matching, captures, iteration
cargo run --example scanner_tokenize  # multi-pattern scanner, UTF-16 offsets

Supported features

Scanner -- multi-pattern matching with result caching, two search strategies (RegSet for short strings, per-regex for long strings), and automatic UTF-16 position mapping. API-compatible with vscode-oniguruma.

Full Oniguruma regex -- every feature from the C engine:

All Perl/Ruby/Python syntax -- (?:...), (?=...), (?!...), (?<=...), (?<!...), (?>...)
Named captures -- (?<name>...), (?'name'...), (?P<name>...)
Backreferences -- \k<name>, \g<name>, relative \g<-1>
Conditionals -- (?(cond)T|F)
Absent expressions -- (?~...)
Unicode properties -- \p{Greek}, \p{Lu}, \p{Emoji} (902 names) from Unicode 17.0 data; C Oniguruma's last snapshot has Unicode 16.0 (ADR-015)
Grapheme clusters -- \X, text segment boundaries \y, \Y
Callouts -- (?{...}), (*FAIL), (*MAX{n}), (*COUNT), (*CMP)
12 syntax modes -- Oniguruma, Ruby, Perl, Perl_NG, Python, Java, Emacs, Grep, GNU, POSIX Basic/Extended, ASIS
Safety limits -- retry, time, stack, subexp call depth (global + per-search)

Performance

How much faster Ferroni is than each engine: the other engine's time divided by Ferroni's, as the geometric mean over each workload. A range spans the two hosts; below 1×, the other engine is faster.

Ferroni compared with	Text processing, shared syntax	Text processing, Oniguruma syntax	Highlighting, portable grammars
Oniguruma (C)	2.2×	1.3–1.4×	3.6–3.7×
Shiki JS	–	–	1.9–2.0×
Onigmo	2.4–2.6×	1.4–1.6× (16 of 17 cases)	11–13×
PCRE2	1.6–1.8× (31 of 32 cases)	1.3–1.4× (16 of 17 cases)	8.7–9.6×
PCRE2 JIT	0.4–0.5× (31 of 32 cases)	0.3× (16 of 17 cases)	3.0–3.2×
fancy-regex	1.5–1.6× (31 of 32 cases)	2.8–3.1× (15 of 17 cases)	35–36×
regex	0.6–0.7× (31 of 32 cases)	–	–

Measured 2026-10-01 on Blacksmith's macOS arm64 (Apple M4 Pro) and Linux x86-64 (AMD EPYC) runners. Text processing runs 49 tasks over HTML, logs, chat with emoji, Markdown, JSON, CSV and source code, from tags and emails to the RFC 5322 address syntax. Highlighting replays the scanner calls Shiki makes for C, Java and PHP documents, grammars every engine can run. Every engine first has to reproduce Oniguruma's results; the note in a cell says how many tasks an engine could run, – marks one it does not cover. These are observations on the measured workloads, not a promise for every pattern or machine. Every task, hosts, versions and raw data.

Text processing

Ferroni is faster than Oniguruma in 28 of 32 tasks with shared syntax, by up to 36x on email extraction, and in 14 of 17 tasks with Oniguruma syntax. Oniguruma is faster on the RFC 5322 and IPv6 expressions, and by up to 30% on a few Oniguruma-syntax tasks.

When patterns fit the narrower syntax of Rust's regex crate, that engine often has the throughput advantage, and PCRE2's JIT is faster still on most tasks. Neither applies full Unicode case folding (straße does not find STRASSE), and neither finds the earliest match among many patterns in one search, as Oniguruma's RegSet does. They are a good choice when their feature set is enough; Ferroni is for workloads that need Oniguruma features and practical scanner speed.

Syntax highlighting

On the Shiki replays every engine can run, Ferroni is 2.7x to 5.7x faster than Oniguruma. Tokenizing whole documents line by line with the complete Shiki grammars, it is 3x to 4x faster on TypeScript, about 10x on Rust and 29x to 32x on CSS; re-scanning the same line, where Ferroni reuses what it learned about that string, between 14x and 77x (exact numbers). Many TextMate grammars use syntax only Oniguruma reads: PCRE2 and Onigmo reject the C++ grammar, and PCRE2 reads the SCSS grammar differently. There Ferroni is 2x to 3x faster than Oniguruma on C++ and 15x to 21x on SCSS. Grammar compilation is mixed: Ferroni compiles the CSS grammar faster, Oniguruma the smaller Rust grammar, and the TypeScript grammar depends on the host. Ferroni's strongest fit is a scanner that compiles a grammar once and uses it repeatedly.

Compilation and memory

Single-pattern compilation is mixed: Oniguruma compiles plain literals and the lookbehind pattern faster, Ferroni the capture-heavy pattern. The regex crate and fancy-regex take 55x to 64x longer for the capture-heavy pattern. In a separate process-level memory measurement, Ferroni and Oniguruma peak in the same approximate range while compiling and scanning a large TypeScript workload. Details and methodology are in the Memory Measurements.

Ferroni keeps benchmark suites separated by purpose:

Internal suite (regression_bench, Rust-only, regression/optimization work):

cargo bench
cargo bench --bench regression_bench

# compare two local baselines
cargo bench --bench regression_bench -- --save-baseline main
# switch to the branch you want to compare
cargo bench --bench regression_bench -- --baseline main

Reference suite (battle_bench, Ferroni vs Oniguruma for publishable numbers):
```
# one-time setup
./scripts/prepare-oniguruma-sources.sh

cargo bench --features ffi --bench battle_bench
```
Exact external input pins for this suite live in benches/battle_inputs.toml.
Memory comparison (process-isolated peak RSS on the large TypeScript scanner workload):
```
./scripts/run-battle-memory.sh
```

HTML report:

open target/criterion/report/index.html

Architecture

Each C source file maps 1:1 to a Rust module (ADR-001):

C File	Rust Module	Purpose
regparse.c	`regparse.rs`	Pattern parser
regcomp.c	`regcomp.rs`	AST-to-bytecode compiler
regexec.c	`regexec.rs`	VM executor
regint.h	`regint.rs`	Internal types and opcodes
oniguruma.h	`oniguruma.rs`	Public types and constants
regenc.c	`regenc.rs`	Encoding trait
regsyntax.c	`regsyntax.rs`	12 syntax definitions
regset.c	`regset.rs`	Multi-regex search (RegSet)
regerror.c	`regerror.rs`	Error messages
regtrav.c	`regtrav.rs`	Capture tree traversal
unicode.c	`unicode/mod.rs`	Unicode tables and segmentation
--	`scanner.rs`	Multi-pattern scanner for syntax highlighting

Compilation pipeline (same as C):

onig_new() -> onig_compile()
  -> onig_parse_tree()     (pattern -> AST)
  -> reduce_string_list()  (merge adjacent strings)
  -> tune_tree()           (6 optimization sub-passes)
  -> compile_tree()        (AST -> VM bytecode)
  -> set_optimize_info()   (extract search strategy)

Scope

Ferroni targets ASCII/UTF-8 workloads. The following are intentionally not included:

27 of 29 encodings -- only ASCII and UTF-8 (ADR-003). UTF-16/32, Shift_JIS, EUC-JP/KR/TW/CN, Big5, GB18030, KOI8-R, CP1251, and ISO-8859-1 to -16 are not ported. TextMate scanning (vscode-oniguruma compiles every pattern as UTF-8) and jq are fully covered; PHP's mbregex in a non-UTF-8 encoding is not.
POSIX/GNU API -- regcomp/regexec/regfree (ADR-012)
C memory management -- replaced by Rust's Drop trait
onig_new_deluxe -- C-specific allocation, use onig_new() instead

Running tests

The test suites, their stack requirements, and the coverage gate are described in CONTRIBUTING.md.

Test parity

Every upstream test that targets UTF-8 is ported 1:1. EUC-JP tests are out of scope (ADR-003).

Upstream file	Encoding	Upstream	Ferroni	Status
test_utf8.c	UTF-8	1,554	1,561	100%
test_back.c	UTF-8	1,225	1,225	100%
test_syntax.c	UTF-8	144	144	100%
test_options.c	UTF-8	47	48	100%
test_regset.c	UTF-8	4	15	100%
testc.c	EUC-JP	785	—	out of scope
C total (UTF-8)		2,974	2,993	100%
vscode-oniguruma tests	UTF-16	15	25	100%

On top of the ported upstream cases, Ferroni adds Rust-native tests for API integration, edge cases, error paths, and coverage gaps. The tree holds 2,402 #[test] functions in total (some compat functions bundle multiple upstream cases, so this is higher than the parity totals above). The count is derived from the tree by scripts/count-tests.sh, which is the single source for the test counts quoted in this README and in CONTRIBUTING.

C Oniguruma has no coverage reporting. Ferroni's test suite is a strict superset of the upstream tests.

Line coverage: 87.7%, gated at 87%. Measured with cargo-llvm-cov by scripts/coverage.sh, which CI runs on every pull request and which fails the build below the gate -- the threshold is defined in that script. Generated Unicode tables, the FFI bindings and the test files themselves are excluded from the measurement. 42 deeply recursive tests are skipped under LLVM instrumentation (stack overflow from coverage bookkeeping) but pass in normal cargo test.

Architecture decision records

ADR	Decision
001	1:1 structural parity with C original
002	Unsafe code policy
003	Encoding scope: ASCII and UTF-8 only
004	C-to-Rust translation patterns
005	Idiomatic Rust API layer
006	Scanner API for TextMate tokenization
007	SIMD-accelerated search via memchr
008	Rust-only optimizations and performance philosophy
009	Dependency philosophy
010	Benchmark strategy
011	Test strategy and C test suite parity
012	POSIX and GNU API not ported
013	Stack overflow mitigation in debug builds
014	Porting bugs: lessons learned
015	Generate Unicode tables from the UCD

Contributing

Contributions are welcome! Please read CONTRIBUTING.md and review the ADRs before submitting a PR.

Acknowledgments

Ferroni is built on the work of K. Kosako and the Oniguruma contributors. The C original powers regex in Ruby, PHP, TextMate, jq, and many other projects. The Scanner API and its test suite are based on vscode-oniguruma by Microsoft and the VS Code team.

Oniguruma ended in April 2025. Ferroni is an independent continuation by Sebastian Software; it is not affiliated with or endorsed by the original author.

License

BSD-2-Clause (same as Oniguruma)

Ferroni is the one Ferramenta tool that is not published under MIT OR Apache-2.0: it is a line-by-line port of Oniguruma's C source, so it stays under the BSD-2-Clause license of the code it is derived from.

More from Ferramenta

Ferramenta — A family of Rust tools.

Engines

Tool	Job
ferriki	Shiki-compatible syntax highlighting
ferromark	Markdown to HTML, sanitized by default
ferrolex	Spell checking for text and code
ferrocat	Translation catalog engine
ferralk	Glob matching and parallel filesystem walking
ferrugo	PDF previews for untrusted files

Applications

Tool	Job
palamedes	Internationalization for TypeScript applications
dalo	Your team's agent setup, versioned like code

ferroni 1.9.0