Part of [Ferramenta](https://ferramenta.dev), a family of Rust tools.
# Ferroni — Oniguruma, continued in Rust
[](https://oss.sebastian-software.com) [](https://crates.io/crates/ferroni)
[](https://docs.rs/ferroni)
[](https://github.com/sebastian-software/ferroni/actions)
[](https://github.com/sebastian-software/ferroni/blob/main/.github/workflows/ci.yml)
[](CONTRIBUTING.md#getting-started)
[](https://github.com/sebastian-software/ferroni/blob/main/LICENSE)
<p align="center">
Evidence, not badges:
<a href="#test-parity">test parity with the C suite</a> ·
<a href="https://github.com/sebastian-software/ferroni/blob/main/.github/workflows/ci.yml">coverage gate enforced in CI</a> ·
<a href="https://ferroni.dev/adr/002-unsafe-code-policy">unsafe code policy</a>
</p>
---
**The regex engine behind TextMate grammars, jq and PHP's mbregex, continued
in memory-safe Rust, with the vscode-oniguruma scanner built in.**
[Oniguruma](https://github.com/kkos/oniguruma) sits underneath a large part of
the text tooling world. TextMate grammars, and with them VS Code, Shiki and most
syntax highlighters, are written in its syntax. jq and PHP's mbregex use it
directly, and Ruby's engine, [Onigmo](https://github.com/k-takata/Onigmo),
started as a fork of it. Its feature set goes well beyond most regex libraries:
variable-length look-behind, conditionals, absent expressions, subexpression
calls, 902 Unicode properties, and 12 syntax modes from Perl to POSIX.
The C project [ended on April 24, 2025](https://github.com/kkos/oniguruma#readme).
Ferroni carries the engine forward in Rust:
- **Same engine, verified.** A line-by-line port that keeps Oniguruma's module
structure and optimization pipeline, not a lookalike. Every upstream UTF-8
test passes ([test parity](#test-parity)).
- **Memory-safe, no C toolchain.** `cargo add ferroni` and build: no bindgen,
no C compiler. C Oniguruma has a long history of memory-safety CVEs; Ferroni
keeps `unsafe` at 0.4%, every block documented in
[ADR-002](https://ferroni.dev/adr/002-unsafe-code-policy).
- **Faster where highlighters spend their time.** Tokenizing real code with
complete TextMate grammars is between 2x and 30x faster than the C original,
and text search is up to 6x faster ([Performance](#performance)).
- **The vscode-oniguruma scanner, built in.** vscode-textmate and Shiki
tokenize through vscode-oniguruma's multi-pattern scanner. Ferroni ships a
[Scanner API](#scanner-api) of the same shape, UTF-16 offsets included, in the
same crate.
## Why Ferroni exists
We build [Ferromark](https://ferromark.dev/), a Markdown engine in Rust, and
[Ardo](https://www.ardo-docs.dev), a documentation framework, and we want the
whole chain in Rust, including syntax highlighting on par with Shiki. That
highlighter is [Ferriki](https://github.com/sebastian-software/ferriki), a
Shiki-compatible engine that tokenizes TextMate grammars with Ferroni's
scanner. Those grammars are written for Oniguruma: they rely on look-behind,
`\G` anchors, backreferences and other features most regex engines skip.
Bindings to the C library would have kept the C code, the C toolchain and its
memory-safety record. A new engine would have broken the grammars in subtle
ways. So we ported Oniguruma itself, line by line, verified the port against
its own test suite, and then tuned the path that highlighters actually take.
Ferroni is not a replacement for Rust's [`regex`](https://crates.io/crates/regex)
crate. When a pattern fits `regex`'s narrower syntax, `regex` is usually
faster. Ferroni is for workloads that need Oniguruma's features and behavior.
## Quick start
Add to your `Cargo.toml`:
```toml
[dependencies]
ferroni = "1"
```
MSRV: Rust 1.94 (enforced in CI).
### Regex
```rust
use ferroni::prelude::*;
fn main() -> Result<(), RegexError> {
let re = Regex::new(r"(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})")?;
let caps = re.captures("Date: 2026-02-12").unwrap();
assert_eq!(caps.get(0).unwrap().as_str(), "2026-02-12");
assert_eq!(caps.name("year").unwrap().as_str(), "2026");
assert_eq!(caps.name("month").unwrap().as_str(), "02");
Ok(())
}
```
### Scanner API
The Scanner matches multiple patterns simultaneously -- the core operation
behind TextMate-based syntax highlighting. Results include UTF-16 position
mapping for direct use with vscode-textmate and Shiki.
```rust
use ferroni::scanner::{Scanner, ScannerFindOptions};
let mut scanner = Scanner::new(&[
r"\b(function|const|let|var)\b", // keywords
r#""[^"]*""#, // strings
r"//.*$", // comments
]).unwrap();
let code = r#"const x = "hello" // greeting"#;
let m = scanner.find_next_match(code, 0, ScannerFindOptions::NONE).unwrap();
assert_eq!(m.index, 0); // pattern 0 matched first ("const")
assert_eq!(m.capture_indices[0].start, 0);
assert_eq!(m.capture_indices[0].end, 5);
```
For fine-grained control, use `RegexBuilder`:
```rust
use ferroni::prelude::*;
let re = Regex::builder(r"hello")
.case_insensitive(true)
.build()
.unwrap();
assert!(re.is_match("Hello World"));
```
<details>
<summary><strong>Low-level C-style API</strong></summary>
The full C-ported API is also available for advanced usage:
```rust
use ferroni::regcomp::onig_new;
use ferroni::regexec::onig_search;
use ferroni::oniguruma::*;
use ferroni::regsyntax::OnigSyntaxOniguruma;
let reg = onig_new(
b"\\d{4}-\\d{2}-\\d{2}",
ONIG_OPTION_NONE,
&ferroni::encodings::utf8::ONIG_ENCODING_UTF8,
&OnigSyntaxOniguruma,
).unwrap();
let input = b"Date: 2026-02-12";
let (result, region) = onig_search(
®, input, input.len(), 0, input.len(),
Some(OnigRegion::new()), ONIG_OPTION_NONE,
);
assert!(result >= 0);
assert_eq!(result, 6); // match starts at byte 6
```
</details>
Both APIs have a runnable example in [`examples/`](examples):
```bash
cargo run --example basic_match # matching, captures, iteration
cargo run --example scanner_tokenize # multi-pattern scanner, UTF-16 offsets
```
## Supported features
**Scanner** -- multi-pattern matching with result caching, two search
strategies (RegSet for short strings, per-regex for long strings), and
automatic UTF-16 position mapping. API-compatible with
[vscode-oniguruma](https://github.com/microsoft/vscode-oniguruma).
**Full Oniguruma regex** -- every feature from the C engine:
- All Perl/Ruby/Python syntax -- `(?:...)`, `(?=...)`, `(?!...)`, `(?<=...)`, `(?<!...)`, `(?>...)`
- Named captures -- `(?<name>...)`, `(?'name'...)`, `(?P<name>...)`
- Backreferences -- `\k<name>`, `\g<name>`, relative `\g<-1>`
- Conditionals -- `(?(cond)T|F)`
- Absent expressions -- `(?~...)`
- Unicode properties -- `\p{Greek}`, `\p{Lu}`, `\p{Emoji}` (902 names) from
Unicode 17.0 data; C Oniguruma's last snapshot has Unicode 16.0
([ADR-015](https://ferroni.dev/adr/015-unicode-data-from-ucd))
- Grapheme clusters -- `\X`, text segment boundaries `\y`, `\Y`
- Callouts -- `(?{...})`, `(*FAIL)`, `(*MAX{n})`, `(*COUNT)`, `(*CMP)`
- 12 syntax modes -- Oniguruma, Ruby, Perl, Perl_NG, Python, Java, Emacs, Grep, GNU, POSIX Basic/Extended, ASIS
- Safety limits -- retry, time, stack, subexp call depth (global + per-search)
## Performance
This summary draws on a recorded
[`battle_bench`](https://github.com/sebastian-software/ferroni/blob/main/benches/battle_bench.rs)
run. Scanner comparisons use complete, unmodified Shiki TextMate grammars for
TypeScript, CSS, and Rust. It shows the direction across the measured
workloads, not a promise for every pattern or machine. Raw results, inputs,
and measurement context are in
[Benchmark Results](https://ferroni.dev/perf/benchmark-results).
### Scanner workloads
For syntax highlighting, Ferroni consistently spends less time finding the next
match and tokenizing lines than Oniguruma, across all three production grammars.
How large the lead is depends on the workload. Tokenizing whole documents line
by line, each line handed to the scanner once as a real tokenizer does, Ferroni
is more than twice as fast on TypeScript, about 10x faster on Rust, and about
30x faster on CSS
([exact numbers](https://ferroni.dev/perf/benchmark-results#scanner-on-whole-documents-line-by-line)).
Re-scanning the same line, where Ferroni can reuse what it learned about that
string, the lead grows to between about 15x and nearly 300x
([exact numbers](https://ferroni.dev/perf/benchmark-results#scanner-with-full-shiki-textmate-grammars)).
Compiling the full TypeScript and CSS grammars is also faster in Ferroni, while
Oniguruma still starts faster on the smaller Rust grammar. In practice,
Ferroni's strongest fit is a scanner that compiles a grammar once and uses it
repeatedly.
### Search and matching
On the measured log-style searches, Ferroni is faster than Oniguruma for both
matches and misses, capture-based extraction, timestamps, and multi-pattern
search. The advantage ranges from modest to several-fold, spread across cases
rather than resting on one standout case. Ferroni also comes out ahead across
the measured Oniguruma-compatible matching patterns, including lookaround,
Unicode properties, backreferences, and alternation.
When patterns fit the narrower syntax of Rust's
[`regex`](https://crates.io/crates/regex) crate, that engine often has the
throughput advantage. It is a good choice for plain-text matching when its
feature set is enough; Ferroni is for workloads that need Oniguruma features
and practical scanner speed.
### Compilation and memory
Compile-time results are mixed. Ferroni compiles the full TypeScript and CSS
grammars faster, but Oniguruma has an edge on Rust grammar startup and on some
individual patterns. The `regex` crate can take longer to compile, especially
for capture-heavy patterns. In a separate process-level memory measurement,
Ferroni and Oniguruma peak in the same approximate range while compiling and
scanning a large TypeScript workload. Details and methodology are in the
[Memory Measurements](https://ferroni.dev/perf/memory-measurements).
<details>
<summary><strong>Running benchmarks</strong></summary>
Ferroni keeps benchmark suites separated by purpose:
- **Internal suite (`regression_bench`, Rust-only, regression/optimization work):**
```bash
cargo bench
cargo bench --bench regression_bench
cargo bench --bench regression_bench -- --save-baseline main
cargo bench --bench regression_bench -- --baseline main
```
- **Reference suite (`battle_bench`, Ferroni vs Oniguruma for publishable numbers):**
```bash
./scripts/prepare-oniguruma-sources.sh
cargo bench --features ffi --bench battle_bench
```
Exact external input pins for this suite live in
[`benches/battle_inputs.toml`](benches/battle_inputs.toml).
- **Memory comparison (process-isolated peak RSS on the large TypeScript scanner workload):**
```bash
./scripts/run-battle-memory.sh
```
- **HTML report:**
```bash
open target/criterion/report/index.html
```
</details>
## Architecture
Each C source file maps 1:1 to a Rust module ([ADR-001](https://ferroni.dev/adr/001-one-to-one-parity-with-c-original)):
| regparse.c | `regparse.rs` | Pattern parser |
| regcomp.c | `regcomp.rs` | AST-to-bytecode compiler |
| regexec.c | `regexec.rs` | VM executor |
| regint.h | `regint.rs` | Internal types and opcodes |
| oniguruma.h | `oniguruma.rs` | Public types and constants |
| regenc.c | `regenc.rs` | Encoding trait |
| regsyntax.c | `regsyntax.rs` | 12 syntax definitions |
| regset.c | `regset.rs` | Multi-regex search (RegSet) |
| regerror.c | `regerror.rs` | Error messages |
| regtrav.c | `regtrav.rs` | Capture tree traversal |
| unicode.c | `unicode/mod.rs` | Unicode tables and segmentation |
| -- | `scanner.rs` | Multi-pattern scanner for syntax highlighting |
**Compilation pipeline** (same as C):
```
onig_new() -> onig_compile()
-> onig_parse_tree() (pattern -> AST)
-> reduce_string_list() (merge adjacent strings)
-> tune_tree() (6 optimization sub-passes)
-> compile_tree() (AST -> VM bytecode)
-> set_optimize_info() (extract search strategy)
```
## Scope
Ferroni targets ASCII/UTF-8 workloads. The following are intentionally not included:
- **27 of 29 encodings** -- only ASCII and UTF-8 ([ADR-003](https://ferroni.dev/adr/003-encoding-scope-ascii-and-utf8-only)).
UTF-16/32, Shift_JIS, EUC-JP/KR/TW/CN, Big5, GB18030, KOI8-R, CP1251, and
ISO-8859-1 to -16 are not ported. TextMate scanning (vscode-oniguruma
compiles every pattern as UTF-8) and jq are fully covered; PHP's mbregex in a
non-UTF-8 encoding is not.
- **POSIX/GNU API** -- `regcomp`/`regexec`/`regfree` ([ADR-012](https://ferroni.dev/adr/012-posix-and-gnu-api-not-ported))
- **C memory management** -- replaced by Rust's `Drop` trait
- **`onig_new_deluxe`** -- C-specific allocation, use `onig_new()` instead
## Running tests
The test suites, their stack requirements, and the coverage gate are described
in [CONTRIBUTING.md](CONTRIBUTING.md#running-tests).
## Test parity
Every upstream test that targets UTF-8 is ported 1:1. EUC-JP tests are
out of scope ([ADR-003](https://ferroni.dev/adr/003-encoding-scope-ascii-and-utf8-only)).
| test_utf8.c | UTF-8 | 1,554 | 1,561 | 100% |
| test_back.c | UTF-8 | 1,225 | 1,225 | 100% |
| test_syntax.c | UTF-8 | 144 | 144 | 100% |
| test_options.c | UTF-8 | 47 | 48 | 100% |
| test_regset.c | UTF-8 | 4 | 15 | 100% |
| testc.c | EUC-JP | 785 | — | out of scope |
| **C total (UTF-8)** | | **2,974** | **2,993** | **100%** |
| [vscode-oniguruma](https://github.com/microsoft/vscode-oniguruma) tests | UTF-16 | 15 | 25 | 100% |
On top of the ported upstream cases, Ferroni adds Rust-native tests for
API integration, edge cases, error paths, and coverage gaps. The tree holds
**2,349 `#[test]` functions** in total (some compat functions bundle multiple
upstream cases, so this is higher than the parity totals above). The count is
derived from the tree by [`scripts/count-tests.sh`](scripts/count-tests.sh),
which is the single source for the test counts quoted in this README and in
CONTRIBUTING.
C Oniguruma has no coverage reporting. Ferroni's test suite is a strict
superset of the upstream tests.
**Line coverage: 87.7%, gated at 87%.** Measured with
[cargo-llvm-cov](https://github.com/taiki-e/cargo-llvm-cov) by
[`scripts/coverage.sh`](scripts/coverage.sh), which CI runs on every pull
request and which fails the build below the gate -- the threshold is defined in
that script. Generated Unicode tables, the FFI bindings and the test files
themselves are excluded from the measurement. 42 deeply recursive tests are
skipped under LLVM instrumentation (stack overflow from coverage bookkeeping)
but pass in normal `cargo test`.
## Architecture decision records
| [001](https://ferroni.dev/adr/001-one-to-one-parity-with-c-original) | 1:1 structural parity with C original |
| [002](https://ferroni.dev/adr/002-unsafe-code-policy) | Unsafe code policy |
| [003](https://ferroni.dev/adr/003-encoding-scope-ascii-and-utf8-only) | Encoding scope: ASCII and UTF-8 only |
| [004](https://ferroni.dev/adr/004-c-to-rust-translation-patterns) | C-to-Rust translation patterns |
| [005](https://ferroni.dev/adr/005-idiomatic-rust-api-layer) | Idiomatic Rust API layer |
| [006](https://ferroni.dev/adr/006-scanner-api) | Scanner API for TextMate tokenization |
| [007](https://ferroni.dev/adr/007-simd-accelerated-search) | SIMD-accelerated search via memchr |
| [008](https://ferroni.dev/adr/008-rust-only-optimizations) | Rust-only optimizations and performance philosophy |
| [009](https://ferroni.dev/adr/009-dependency-philosophy) | Dependency philosophy |
| [010](https://ferroni.dev/adr/010-benchmark-strategy) | Benchmark strategy |
| [011](https://ferroni.dev/adr/011-test-strategy-and-c-test-parity) | Test strategy and C test suite parity |
| [012](https://ferroni.dev/adr/012-posix-and-gnu-api-not-ported) | POSIX and GNU API not ported |
| [013](https://ferroni.dev/adr/013-stack-overflow-debug-builds) | Stack overflow mitigation in debug builds |
| [014](https://ferroni.dev/adr/014-porting-bugs-lessons-learned) | Porting bugs: lessons learned |
| [015](https://ferroni.dev/adr/015-unicode-data-from-ucd) | Generate Unicode tables from the UCD |
## Contributing
Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md)
and review the ADRs before submitting a PR.
## Acknowledgments
Ferroni is built on the work of [K. Kosako](https://github.com/kkos) and
the Oniguruma contributors. The C original powers regex in
[Ruby](https://www.ruby-lang.org/), [PHP](https://www.php.net/),
[TextMate](https://macromates.com/), [jq](https://jqlang.github.io/jq/),
and many other projects. The Scanner API and its test suite are based on
[vscode-oniguruma](https://github.com/microsoft/vscode-oniguruma)
by Microsoft and the VS Code team.
Oniguruma ended in April 2025. Ferroni is an independent continuation by
Sebastian Software; it is not affiliated with or endorsed by the original
author.
## License
[BSD-2-Clause](LICENSE) (same as Oniguruma)
Ferroni is the one Ferramenta tool that is not published under `MIT OR
Apache-2.0`: it is a line-by-line port of Oniguruma's C source, so it stays
under the BSD-2-Clause license of the code it is derived from.
### <a href="https://ferramenta.dev"><img src="https://raw.githubusercontent.com/sebastian-software/ferramenta/main/app/assets/brand/logo-light.svg" width="24" height="24" alt="" /> More from Ferramenta</a>
[Ferramenta](https://ferramenta.dev) — A family of Rust tools.
**The content pipeline**
| [ferriki](https://github.com/sebastian-software/ferriki) | Shiki-compatible syntax highlighting |
| [ferromark](https://sebastian-software.github.io/ferromark/) | Markdown to HTML, sanitized by default |
**The language workshop**
| [ferrolex](https://github.com/sebastian-software/ferrolex) | Spell checking for text and code |
| [ferrocat](https://ferrocat.dev) | Translation catalog engine |
| [palamedes](https://palamedes.dev) | Internationalization for TypeScript applications |
**On the workbench**
| [ferralk](https://github.com/sebastian-software/ferralk) | Glob matching and parallel filesystem walking |
| [ferrugo](https://github.com/sebastian-software/ferrugo) | PDF previews for untrusted files |
| [dalo](https://dalo.sh) | Team agent skills, versioned and synced as code |
| [cuttledoc](https://github.com/sebastian-software/cuttledoc) | Local-first speech transcription |
---
<p align="center">
<a href="https://oss.sebastian-software.com"><img src="https://raw.githubusercontent.com/sebastian-software/oss.sebastian-software.com/main/app/assets/logo-software.svg" alt="Sebastian Software" width="160" /></a><br />
TypeScript, React & Rust consulting<br />
Experts in Agentic Software Development<br />
<a href="https://sebastian-software.de">Work with us</a> · <a href="https://oss.sebastian-software.com">More open source</a>
</p>
<p align="center">Copyright © 2026 Sebastian Software GmbH</p>