regexr

A specialized, pure-Rust regex engine designed for LLM tokenization and complex pattern matching.

Originally created as the regex backend for splintr, an LLM tokenizer. Passes compliance tests for industry-standard tokenizer patterns (OpenAI's cl100k_base, Meta's Llama 3).

Please report issues on the Issue Tracker.

🎯 When to use regexr

This is a specialized tool, not a general-purpose replacement.

The Rust ecosystem already has the excellent, battle-tested regex crate. For 99% of use cases, you should use that.

Only use regexr if you specifically need:

1. Lookarounds: You need features like (?=...), (?<=...), or (?!\S) without C dependencies.
   • Why not regex? It intentionally omits lookarounds to guarantee linear time.
   • Why not pcre2? Requires C library and FFI.
2. JIT Compilation in Pure Rust: You want native code generation for hot patterns without C dependencies.
   • Why not regex/fancy-regex? Neither offers JIT compilation.
   • Why not pcre2? Requires C library and FFI.
3. Pure Rust Dependency: You need advanced features (Lookarounds, Backreferences) but cannot use pcre2 due to unsafe C bindings or build complexity.
4. Bounded Execution: You want ReDoS protection that memoizes states (guaranteeing completion) rather than just aborting after a timeout (like pcre2).

The Problem Solved

Developers building LLM tokenizers (like GPT-4 or Llama 3) currently face a dilemma in Rust:

• regex crate: Fast, safe, but lacks lookarounds and JIT compilation.
• fancy-regex: Supports lookarounds, but lacks JIT compilation.
• pcre2: Supports everything including JIT, but introduces unsafe C bindings and external dependencies.

regexr bridges this gap. It provides Lookarounds + JIT compilation + Backreferences while remaining 100% Pure Rust.

Installation

Add this to your Cargo.toml:

[dependencies]
regexr = "0.x"

For JIT compilation support:

[dependencies]
regexr = { version = "0.x", features = ["full"] }

Usage

Basic matching

use regexr::Regex;

let re = Regex::new(r"\w+").unwrap();
assert!(re.is_match("hello"));

// Find first match
if let Some(m) = re.find("hello world") {
    println!("Found: {}", m.as_str()); // "hello"
}

// Find all matches
for m in re.find_iter("hello world") {
    println!("{}", m.as_str());
}

Capture groups

use regexr::Regex;

let re = Regex::new(r"(\w+)@(\w+)\.(\w+)").unwrap();
let caps = re.captures("user@example.com").unwrap();

println!("{}", &caps[0]); // "user@example.com"
println!("{}", &caps[1]); // "user"
println!("{}", &caps[2]); // "example"
println!("{}", &caps[3]); // "com"

Named captures

use regexr::Regex;

let re = Regex::new(r"(?P<user>\w+)@(?P<domain>\w+\.\w+)").unwrap();
let caps = re.captures("user@example.com").unwrap();

println!("{}", &caps["user"]);   // "user"
println!("{}", &caps["domain"]); // "example.com"

JIT compilation

Enable JIT for patterns that will be matched many times:

use regexr::RegexBuilder;

let re = RegexBuilder::new(r"\w+")
    .jit(true)
    .build()
    .unwrap();

assert!(re.is_match("hello"));

Prefix optimization for tokenizers

For patterns with many literal alternatives (e.g., keyword matching in tokenizers):

use regexr::RegexBuilder;

let re = RegexBuilder::new(r"(function|for|while|if|else|return)")
    .optimize_prefixes(true)
    .build()
    .unwrap();

assert!(re.is_match("function"));

Text replacement

use regexr::Regex;

let re = Regex::new(r"\d+").unwrap();

// Replace first match
let result = re.replace("abc 123 def", "NUM");
assert_eq!(result, "abc NUM def");

// Replace all matches
let result = re.replace_all("abc 123 def 456", "NUM");
assert_eq!(result, "abc NUM def NUM");

Feature Flags

• simd (default): Enables SIMD-accelerated literal search
• jit: Enables JIT compilation (x86-64 and ARM64)
• full: Enables both JIT and SIMD

Platform Support

Build without default features for a minimal installation (also works for WASM):

cargo build --no-default-features                           # Minimal (PikeVM + LazyDFA only)
cargo build --no-default-features --target wasm32-unknown-unknown  # WASM target

Build with all optimizations:

cargo build --features "full"

Engine Selection

The library automatically selects the best execution engine based on pattern characteristics:

Non-JIT mode (default):

• ShiftOr: Small patterns (≤64 states) without anchors/word boundaries
• EagerDfa: Patterns with word boundaries or anchors
• LazyDfa: General patterns with on-demand state construction
• BacktrackingVm: Patterns with backreferences
• PikeVm: Patterns with lookaround or non-greedy quantifiers

JIT mode (with jit feature):

• BacktrackingJit: Patterns with backreferences
• TaggedNfa: Patterns with lookaround or non-greedy quantifiers
• JitShiftOr: Small patterns with alternations
• DFA JIT: General patterns, benefits from SIMD prefiltering

See docs/architecture.md for details on the engine selection logic.

Performance

Speedup relative to regex crate (higher is better):

Highlights (speedup vs regex crate):

• regexr-jit excels at log parsing (4x faster than regex)
• regexr (non-JIT) matches regex performance on most patterns
• Both outperform fancy-regex and pcre2 (non-JIT) consistently

Documentation

• Architecture Overview - Engine architecture and selection logic
• Features - Detailed feature documentation

Citation

If you use regexr in your research, please cite:

@software{regexr2025,
  author       = {Syah, Farhan},
  title        = {regexr: A Pure-Rust Regex Engine with JIT Compilation for LLM Tokenization},
  year         = {2025},
  url          = {https://github.com/ml-rust/regexr},
  note         = {Pure-Rust regex engine with lookaround support and JIT compilation}
}