Part of [Ferramenta](https://ferramenta.dev), a family of Rust tools.
# Ferroni — Oniguruma, continued in Rust
[](https://oss.sebastian-software.com) [](https://crates.io/crates/ferroni)
[](https://docs.rs/ferroni)
[](https://github.com/sebastian-software/ferroni/actions)
[](https://github.com/sebastian-software/ferroni/blob/main/.github/workflows/ci.yml)
[](CONTRIBUTING.md#getting-started)
[](https://github.com/sebastian-software/ferroni/blob/main/LICENSE)
<p align="center">
Evidence, not badges:
<a href="#test-parity">test parity with the C suite</a> ·
<a href="https://github.com/sebastian-software/ferroni/blob/main/.github/workflows/ci.yml">coverage gate enforced in CI</a> ·
<a href="https://ferroni.dev/adr/002-unsafe-code-policy">unsafe code policy</a>
</p>
---
**The regex engine behind TextMate grammars, jq and PHP's mbregex, continued
in memory-safe Rust, with the vscode-oniguruma scanner built in.**
[Oniguruma](https://github.com/kkos/oniguruma) sits underneath a large part of
the text tooling world. TextMate grammars, and with them VS Code, Shiki and most
syntax highlighters, are written in its syntax. jq and PHP's mbregex use it
directly, and Ruby's engine, [Onigmo](https://github.com/k-takata/Onigmo),
started as a fork of it. Its feature set goes well beyond most regex libraries:
variable-length look-behind, conditionals, absent expressions, subexpression
calls, 902 Unicode properties, and 12 syntax modes from Perl to POSIX.
The C project [ended on April 24, 2025](https://github.com/kkos/oniguruma#readme).
Ferroni carries the engine forward in Rust:
- **Same engine, verified.** A line-by-line port that keeps Oniguruma's module
structure and optimization pipeline, not a lookalike. Every upstream UTF-8
test passes ([test parity](#test-parity)).
- **Memory-safe, no C toolchain.** `cargo add ferroni` and build: no bindgen,
no C compiler. C Oniguruma has a long history of memory-safety CVEs; Ferroni
keeps `unsafe` at 0.4%, every block documented in
[ADR-002](https://ferroni.dev/adr/002-unsafe-code-policy).
- **Faster where highlighters spend their time.** Tokenizing real code with
complete TextMate grammars is between 2x and 30x faster than the C original,
and text search is up to 6x faster ([Performance](#performance)).
- **The vscode-oniguruma scanner, built in.** vscode-textmate and Shiki
tokenize through vscode-oniguruma's multi-pattern scanner. Ferroni ships a
[Scanner API](#scanner-api) of the same shape, UTF-16 offsets included, in the
same crate.
## Why Ferroni exists
We build [Ferromark](https://ferromark.dev/), a Markdown engine in Rust, and
[Ardo](https://www.ardo-docs.dev), a documentation framework, and we want the
whole chain in Rust, including syntax highlighting on par with Shiki. That
highlighter is [Ferriki](https://github.com/sebastian-software/ferriki), a
Shiki-compatible engine that tokenizes TextMate grammars with Ferroni's
scanner. Those grammars are written for Oniguruma: they rely on look-behind,
`\G` anchors, backreferences and other features most regex engines skip.
Bindings to the C library would have kept the C code, the C toolchain and its
memory-safety record. A new engine would have broken the grammars in subtle
ways. So we ported Oniguruma itself, line by line, verified the port against
its own test suite, and then tuned the path that highlighters actually take.
Ferroni is not a replacement for Rust's [`regex`](https://crates.io/crates/regex)
crate. When a pattern fits `regex`'s narrower syntax, `regex` is usually
faster. Ferroni is for workloads that need Oniguruma's features and behavior.
## Quick start
Add to your `Cargo.toml`:
```toml
[dependencies]
ferroni = "1"
```
MSRV: Rust 1.94 (enforced in CI).
### Regex
```rust
use ferroni::prelude::*;
fn main() -> Result<(), RegexError> {
let re = Regex::new(r"(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})")?;
let caps = re.captures("Date: 2026-02-12").unwrap();
assert_eq!(caps.get(0).unwrap().as_str(), "2026-02-12");
assert_eq!(caps.name("year").unwrap().as_str(), "2026");
assert_eq!(caps.name("month").unwrap().as_str(), "02");
Ok(())
}
```
### Scanner API
The Scanner matches multiple patterns simultaneously -- the core operation
behind TextMate-based syntax highlighting. Results include UTF-16 position
mapping for direct use with vscode-textmate and Shiki.
```rust
use ferroni::scanner::{Scanner, ScannerFindOptions};
let mut scanner = Scanner::new(&[
r"\b(function|const|let|var)\b", // keywords
r#""[^"]*""#, // strings
r"//.*$", // comments
]).unwrap();
let code = r#"const x = "hello" // greeting"#;
let m = scanner.find_next_match(code, 0, ScannerFindOptions::NONE).unwrap();
assert_eq!(m.index, 0); // pattern 0 matched first ("const")
assert_eq!(m.capture_indices[0].start, 0);
assert_eq!(m.capture_indices[0].end, 5);
```
A grammar builds one scanner per rule context, and the same patterns recur
across them. Build them with `Scanner::with_pattern_cache` and one
`ScannerPatternCache`, and each distinct pattern is compiled once and shared;
the cache is yours to keep or drop, with no process-wide state
([ADR-006](https://ferroni.dev/adr/006-scanner-api)).
For fine-grained control, use `RegexBuilder`:
```rust
use ferroni::prelude::*;
let re = Regex::builder(r"hello")
.case_insensitive(true)
.build()
.unwrap();
assert!(re.is_match("Hello World"));
```
<details>
<summary><strong>Low-level C-style API</strong></summary>
The full C-ported API is also available for advanced usage:
```rust
use ferroni::regcomp::onig_new;
use ferroni::regexec::onig_search;
use ferroni::oniguruma::*;
use ferroni::regsyntax::OnigSyntaxOniguruma;
let reg = onig_new(
b"\\d{4}-\\d{2}-\\d{2}",
ONIG_OPTION_NONE,
&ferroni::encodings::utf8::ONIG_ENCODING_UTF8,
&OnigSyntaxOniguruma,
).unwrap();
let input = b"Date: 2026-02-12";
let (result, region) = onig_search(
®, input, input.len(), 0, input.len(),
Some(OnigRegion::new()), ONIG_OPTION_NONE,
);
assert!(result >= 0);
assert_eq!(result, 6); // match starts at byte 6
```
</details>
Both APIs have a runnable example in [`examples/`](examples):
```bash
cargo run --example basic_match # matching, captures, iteration
cargo run --example scanner_tokenize # multi-pattern scanner, UTF-16 offsets
```
## Supported features
**Scanner** -- multi-pattern matching with result caching, two search
strategies (RegSet for short strings, per-regex for long strings), and
automatic UTF-16 position mapping. API-compatible with
[vscode-oniguruma](https://github.com/microsoft/vscode-oniguruma).
**Full Oniguruma regex** -- every feature from the C engine:
- All Perl/Ruby/Python syntax -- `(?:...)`, `(?=...)`, `(?!...)`, `(?<=...)`, `(?<!...)`, `(?>...)`
- Named captures -- `(?<name>...)`, `(?'name'...)`, `(?P<name>...)`
- Backreferences -- `\k<name>`, `\g<name>`, relative `\g<-1>`
- Conditionals -- `(?(cond)T|F)`
- Absent expressions -- `(?~...)`
- Unicode properties -- `\p{Greek}`, `\p{Lu}`, `\p{Emoji}` (902 names) from
Unicode 17.0 data; C Oniguruma's last snapshot has Unicode 16.0
([ADR-015](https://ferroni.dev/adr/015-unicode-data-from-ucd))
- Grapheme clusters -- `\X`, text segment boundaries `\y`, `\Y`
- Callouts -- `(?{...})`, `(*FAIL)`, `(*MAX{n})`, `(*COUNT)`, `(*CMP)`
- 12 syntax modes -- Oniguruma, Ruby, Perl, Perl_NG, Python, Java, Emacs, Grep, GNU, POSIX Basic/Extended, ASIS
- Safety limits -- retry, time, stack, subexp call depth (global + per-search)
## Performance
How much faster Ferroni is than each engine: the other engine's time divided
by Ferroni's, as the geometric mean over each workload. A range spans the two
hosts; below 1×, the other engine is faster.
| Oniguruma (C) | 2.2× | 1.3–1.4× | 3.6–3.7× |
| Shiki JS | – | – | 1.9–2.0× |
| Onigmo | 2.4–2.6× | 1.4–1.6× (16 of 17 cases) | 11–13× |
| PCRE2 | 1.6–1.8× (31 of 32 cases) | 1.3–1.4× (16 of 17 cases) | 8.7–9.6× |
| PCRE2 JIT | 0.4–0.5× (31 of 32 cases) | 0.3× (16 of 17 cases) | 3.0–3.2× |
| fancy-regex | 1.5–1.6× (31 of 32 cases) | 2.8–3.1× (15 of 17 cases) | 35–36× |
| regex | 0.6–0.7× (31 of 32 cases) | – | – |
Measured 2026-10-01 on Blacksmith's macOS arm64 (Apple M4 Pro) and Linux
x86-64 (AMD EPYC) runners. Text processing runs 49 tasks over HTML, logs, chat
with emoji, Markdown, JSON, CSV and source code, from tags and emails to the
RFC 5322 address syntax. Highlighting replays the scanner calls Shiki makes for
C, Java and PHP documents, grammars every engine can run. Every engine first
has to reproduce Oniguruma's results; the note in a cell says how many tasks
an engine could run, `–` marks one it does not cover. These are observations on
the measured workloads, not a promise for every pattern or machine.
[Every task, hosts, versions and raw data](https://ferroni.dev/perf/engine-comparison).
### Text processing
Ferroni is faster than Oniguruma in 28 of 32 tasks with shared syntax, by up to
36x on email extraction, and in 14 of 17 tasks with Oniguruma syntax. Oniguruma
is faster on the RFC 5322 and IPv6 expressions, and by up to 30% on a few
Oniguruma-syntax tasks.
When patterns fit the narrower syntax of Rust's
[`regex`](https://crates.io/crates/regex) crate, that engine often has the
throughput advantage, and PCRE2's JIT is faster still on most tasks. Neither
applies full Unicode case folding (`straße` does not find `STRASSE`), and
neither finds the earliest match among many patterns in one search, as
Oniguruma's RegSet does. They are a good choice when their feature set is
enough; Ferroni is for workloads that need Oniguruma features and practical
scanner speed.
### Syntax highlighting
On the Shiki replays every engine can run, Ferroni is 2.7x to 5.7x faster than
Oniguruma. Tokenizing whole documents line by line with the complete Shiki
grammars, it is 3x to 4x faster on TypeScript, about 10x on Rust and 29x to 32x on
CSS; re-scanning the same line, where Ferroni reuses what it learned about that
string, between 14x and 77x
([exact numbers](https://ferroni.dev/perf/engine-comparison#highlighting)).
Many TextMate grammars use syntax only Oniguruma reads: PCRE2 and Onigmo reject
the C++ grammar, and PCRE2 reads the SCSS grammar differently. There Ferroni is
2x to 3x faster than Oniguruma on C++ and 15x to 21x on SCSS. Grammar
compilation is mixed: Ferroni compiles the CSS grammar faster, Oniguruma the
smaller Rust grammar, and the TypeScript grammar depends on the host. Ferroni's
strongest fit is a scanner that compiles a grammar once and uses it repeatedly.
### Compilation and memory
Single-pattern compilation is mixed: Oniguruma compiles plain literals and the
lookbehind pattern faster, Ferroni the capture-heavy pattern. The `regex` crate
and fancy-regex take 55x to 64x longer for the capture-heavy pattern. In a
separate process-level memory measurement, Ferroni and Oniguruma peak in the
same approximate range while compiling and scanning a large TypeScript
workload. Details and methodology are in the
[Memory Measurements](https://ferroni.dev/perf/memory-measurements).
<details>
<summary><strong>Running benchmarks</strong></summary>
Ferroni keeps benchmark suites separated by purpose:
- **Internal suite (`regression_bench`, Rust-only, regression/optimization work):**
```bash
cargo bench
cargo bench --bench regression_bench
cargo bench --bench regression_bench -- --save-baseline main
cargo bench --bench regression_bench -- --baseline main
```
- **Reference suite (`battle_bench`, Ferroni vs Oniguruma for publishable numbers):**
```bash
./scripts/prepare-oniguruma-sources.sh
cargo bench --features ffi --bench battle_bench
```
Exact external input pins for this suite live in
[`benches/battle_inputs.toml`](benches/battle_inputs.toml).
- **Memory comparison (process-isolated peak RSS on the large TypeScript scanner workload):**
```bash
./scripts/run-battle-memory.sh
```
- **HTML report:**
```bash
open target/criterion/report/index.html
```
</details>
## Architecture
Each C source file maps 1:1 to a Rust module ([ADR-001](https://ferroni.dev/adr/001-one-to-one-parity-with-c-original)):
| regparse.c | `regparse.rs` | Pattern parser |
| regcomp.c | `regcomp.rs` | AST-to-bytecode compiler |
| regexec.c | `regexec.rs` | VM executor |
| regint.h | `regint.rs` | Internal types and opcodes |
| oniguruma.h | `oniguruma.rs` | Public types and constants |
| regenc.c | `regenc.rs` | Encoding trait |
| regsyntax.c | `regsyntax.rs` | 12 syntax definitions |
| regset.c | `regset.rs` | Multi-regex search (RegSet) |
| regerror.c | `regerror.rs` | Error messages |
| regtrav.c | `regtrav.rs` | Capture tree traversal |
| unicode.c | `unicode/mod.rs` | Unicode tables and segmentation |
| -- | `scanner.rs` | Multi-pattern scanner for syntax highlighting |
**Compilation pipeline** (same as C):
```
onig_new() -> onig_compile()
-> onig_parse_tree() (pattern -> AST)
-> reduce_string_list() (merge adjacent strings)
-> tune_tree() (6 optimization sub-passes)
-> compile_tree() (AST -> VM bytecode)
-> set_optimize_info() (extract search strategy)
```
## Scope
Ferroni targets ASCII/UTF-8 workloads. The following are intentionally not included:
- **27 of 29 encodings** -- only ASCII and UTF-8 ([ADR-003](https://ferroni.dev/adr/003-encoding-scope-ascii-and-utf8-only)).
UTF-16/32, Shift_JIS, EUC-JP/KR/TW/CN, Big5, GB18030, KOI8-R, CP1251, and
ISO-8859-1 to -16 are not ported. TextMate scanning (vscode-oniguruma
compiles every pattern as UTF-8) and jq are fully covered; PHP's mbregex in a
non-UTF-8 encoding is not.
- **POSIX/GNU API** -- `regcomp`/`regexec`/`regfree` ([ADR-012](https://ferroni.dev/adr/012-posix-and-gnu-api-not-ported))
- **C memory management** -- replaced by Rust's `Drop` trait
- **`onig_new_deluxe`** -- C-specific allocation, use `onig_new()` instead
## Running tests
The test suites, their stack requirements, and the coverage gate are described
in [CONTRIBUTING.md](CONTRIBUTING.md#running-tests).
## Test parity
Every upstream test that targets UTF-8 is ported 1:1. EUC-JP tests are
out of scope ([ADR-003](https://ferroni.dev/adr/003-encoding-scope-ascii-and-utf8-only)).
| test_utf8.c | UTF-8 | 1,554 | 1,561 | 100% |
| test_back.c | UTF-8 | 1,225 | 1,225 | 100% |
| test_syntax.c | UTF-8 | 144 | 144 | 100% |
| test_options.c | UTF-8 | 47 | 48 | 100% |
| test_regset.c | UTF-8 | 4 | 15 | 100% |
| testc.c | EUC-JP | 785 | — | out of scope |
| **C total (UTF-8)** | | **2,974** | **2,993** | **100%** |
| [vscode-oniguruma](https://github.com/microsoft/vscode-oniguruma) tests | UTF-16 | 15 | 25 | 100% |
On top of the ported upstream cases, Ferroni adds Rust-native tests for
API integration, edge cases, error paths, and coverage gaps. The tree holds
**2,402 `#[test]` functions** in total (some compat functions bundle multiple
upstream cases, so this is higher than the parity totals above). The count is
derived from the tree by [`scripts/count-tests.sh`](scripts/count-tests.sh),
which is the single source for the test counts quoted in this README and in
CONTRIBUTING.
C Oniguruma has no coverage reporting. Ferroni's test suite is a strict
superset of the upstream tests.
**Line coverage: 87.7%, gated at 87%.** Measured with
[cargo-llvm-cov](https://github.com/taiki-e/cargo-llvm-cov) by
[`scripts/coverage.sh`](scripts/coverage.sh), which CI runs on every pull
request and which fails the build below the gate -- the threshold is defined in
that script. Generated Unicode tables, the FFI bindings and the test files
themselves are excluded from the measurement. 42 deeply recursive tests are
skipped under LLVM instrumentation (stack overflow from coverage bookkeeping)
but pass in normal `cargo test`.
## Architecture decision records
| [001](https://ferroni.dev/adr/001-one-to-one-parity-with-c-original) | 1:1 structural parity with C original |
| [002](https://ferroni.dev/adr/002-unsafe-code-policy) | Unsafe code policy |
| [003](https://ferroni.dev/adr/003-encoding-scope-ascii-and-utf8-only) | Encoding scope: ASCII and UTF-8 only |
| [004](https://ferroni.dev/adr/004-c-to-rust-translation-patterns) | C-to-Rust translation patterns |
| [005](https://ferroni.dev/adr/005-idiomatic-rust-api-layer) | Idiomatic Rust API layer |
| [006](https://ferroni.dev/adr/006-scanner-api) | Scanner API for TextMate tokenization |
| [007](https://ferroni.dev/adr/007-simd-accelerated-search) | SIMD-accelerated search via memchr |
| [008](https://ferroni.dev/adr/008-rust-only-optimizations) | Rust-only optimizations and performance philosophy |
| [009](https://ferroni.dev/adr/009-dependency-philosophy) | Dependency philosophy |
| [010](https://ferroni.dev/adr/010-benchmark-strategy) | Benchmark strategy |
| [011](https://ferroni.dev/adr/011-test-strategy-and-c-test-parity) | Test strategy and C test suite parity |
| [012](https://ferroni.dev/adr/012-posix-and-gnu-api-not-ported) | POSIX and GNU API not ported |
| [013](https://ferroni.dev/adr/013-stack-overflow-debug-builds) | Stack overflow mitigation in debug builds |
| [014](https://ferroni.dev/adr/014-porting-bugs-lessons-learned) | Porting bugs: lessons learned |
| [015](https://ferroni.dev/adr/015-unicode-data-from-ucd) | Generate Unicode tables from the UCD |
## Contributing
Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md)
and review the ADRs before submitting a PR.
## Acknowledgments
Ferroni is built on the work of [K. Kosako](https://github.com/kkos) and
the Oniguruma contributors. The C original powers regex in
[Ruby](https://www.ruby-lang.org/), [PHP](https://www.php.net/),
[TextMate](https://macromates.com/), [jq](https://jqlang.github.io/jq/),
and many other projects. The Scanner API and its test suite are based on
[vscode-oniguruma](https://github.com/microsoft/vscode-oniguruma)
by Microsoft and the VS Code team.
Oniguruma ended in April 2025. Ferroni is an independent continuation by
Sebastian Software; it is not affiliated with or endorsed by the original
author.
## License
[BSD-2-Clause](LICENSE) (same as Oniguruma)
Ferroni is the one Ferramenta tool that is not published under `MIT OR
Apache-2.0`: it is a line-by-line port of Oniguruma's C source, so it stays
under the BSD-2-Clause license of the code it is derived from.
### <a href="https://ferramenta.dev"><img src="https://raw.githubusercontent.com/sebastian-software/ferramenta/main/app/assets/brand/logo-light.svg" width="24" height="24" alt="" /> More from Ferramenta</a>
[Ferramenta](https://ferramenta.dev) — A family of Rust tools.
**Engines**
| [ferriki](https://ferriki.dev) | Shiki-compatible syntax highlighting |
| [ferromark](https://ferromark.dev) | Markdown to HTML, sanitized by default |
| [ferrolex](https://github.com/sebastian-software/ferrolex) | Spell checking for text and code |
| [ferrocat](https://ferrocat.dev) | Translation catalog engine |
| [ferralk](https://github.com/sebastian-software/ferralk) | Glob matching and parallel filesystem walking |
| [ferrugo](https://github.com/sebastian-software/ferrugo) | PDF previews for untrusted files |
**Applications**
| [palamedes](https://palamedes.dev) | Internationalization for TypeScript applications |
| [dalo](https://dalo.sh) | Your team's agent setup, versioned like code |
---
<p align="center">
<a href="https://oss.sebastian-software.com"><img src="https://raw.githubusercontent.com/sebastian-software/oss.sebastian-software.com/main/app/assets/logo-software.svg" alt="Sebastian Software" width="160" /></a><br />
TypeScript, React & Rust consulting<br />
Experts in Agentic Software Development<br />
<a href="https://sebastian-software.de">Work with us</a> · <a href="https://oss.sebastian-software.com">More open source</a>
</p>
<p align="center">Copyright © 2026 Sebastian Software GmbH</p>