TREX (Token Regular EXpression)

A regex-shaped pattern language over typed tokens.

Documentation | Getting started | Pattern syntax | Command reference

The lexer reads the input once into typed tokens and a pattern matches over them: \N is one whole number, \E one email address, \B(...) one balanced bracket group, and the whitespace between tokens is never written. A capture binds a named register that a later =name must equal. Matching does not backtrack. trex is a command-line binary, a Rust library, a Python module and a PowerShell module.

• Features
• Quick start
• Why trex
• What a pattern can say
• Commands
• Architecture
• Performance
• Repository layout
• Building and testing
• Platforms and MSRV
• Wiki
• Credits and influences
• Use of AI tools
• License

Features

• 25 built-in token kinds, from numbers and words to IP addresses, URLs, timestamps and payment cards, each one atom.
• Value predicates in a kind's own units: \N{500..599}, \I{in:10.0.0.0/8}, \T{age<24h}.
• Balanced groups and named registers, with back-references exact or up to case, shape, representation or a typed relation.
• Custom atoms declared in a file with test lines, beside a shipped library of 75 named kinds and patterns.
• Rewrites that slice a capture by its typed fields, and redaction that keeps named fields.
• Files, standard input and directories walked under .gitignore rules, with in-place rewriting behind a diff dry run.
• head, tail and lines, and the same windows on a scan, a rewrite or a redaction, read from the end of a file they sit at; --follow reads on as the file grows.
• Ten property axes, read by a command over the token stream or as a term inside a pattern.
• A CUDA backend in the default build for patterns that read only token kinds.
• A PowerShell module of 49 cmdlets that write matches, findings and reports as objects, with each register's value as a .NET type.

Quick start

Install the command from crates.io, where the package is trex-re; the command it installs is trex:

cargo install trex-re

or download the trex binary for Windows x64, Linux x64, FreeBSD x64 or macOS arm64 from the latest release.

A command reads files, directories, - for standard input, or --text:

$ trex scan '\E:e' --text 'ping bob@x.com' --json
[{"start":5,"end":14,"text":"bob@x.com","captures":{"e":"bob@x.com"}}]

$ trex rewrite '\E:e' '[${e:domain}]' --text 'mail bob@x.com now'
mail [x.com] now

$ trex redact '\E:e' --keep e:domain --text 'mail bob@x.com now'
mail ****x.com now

A directory is walked with hidden and binary files skipped, each match carries its path, line and column, and -C prints the lines around it. With logs/a.log holding the five lines alpha 10, beta 20, gamma 300, delta 4000 and epsilon 5:

$ trex scan '\N{>=1000}' logs/ -C 1
logs/a.log-3-gamma 300
logs/a.log:4:7: "4000"
logs/a.log-5-epsilon 5

As a library, the package is trex-re and the crate it names is trex:

[dependencies]
trex-re = "0.2.0"

The Python module installs from PyPI:

pip install trex-re

>>> import trex
>>> m = trex.Pattern(r"<\W:t>.*</=t>").find("say <div>hi</div> now")
>>> (m.start, m.end, m.text, m["t"])
(4, 17, '<div>hi</div>', 'div')

The PowerShell module installs from the PowerShell Gallery:

Install-Module Trex

PS> Select-TrexMatch '\R:t' -InputObject 'retried after 1500ms' | ForEach-Object { $_.Groups[0].Value.TotalSeconds }
1.5

PS> 'mail bob@x.com now' | Protect-TrexText '\E:e' -Keep e:domain
mail ****x.com now

Why trex

A regular expression reads bytes, so a number is written [-+]?\d+(?:\.\d+)?, the space between two items is \s*, and a capture is numbered by where its parenthesis falls. trex reads tokens. The lexer decides where a number ends, whitespace does not appear in a pattern, and a capture has a name, so adding a group renumbers nothing.

A pattern can also require what a regular expression cannot. Here the closing tag must equal the opening one:

$ trex scan '<\W:t>.*</=t>' --text '<div>hi</div>'
[0..13] "<div>hi</div>"  captures: t="div"

\W:t binds the tag's word to the register t, and =t requires a later token equal to it. The engine keeps registers per thread instead of backtracking. With a bind one token wide, scan time grew at 0.91x to 1.06x of linear from 25,000 to 6,400,000 tokens. A bind of unbounded width, \W+:x =x, grew quadratically from 500 to 8,000 tokens. examples/backref_scaling.rs runs both, and what regex has and trex spells differently maps each regex construct.

What a pattern can say

The construct families, each with the section that defines it. examples/patterns.rs runs each one against a real input and asserts its match count.

Commands

Sixteen pattern tools and ten analysis commands. Every flag is in the command reference.

A build with --features compress adds compress, which reports the code length a context-mixing model assigns to the input.

Architecture

flowchart LR
    subgraph lex ["one lex"]
        Bytes["input<br/>files, stdin, --text"]
        Enc["transcode<br/>UTF-8, UTF-16, UTF-32"]
        Lexer["lexer<br/>typed tokens, bracket mates"]
        Bytes --> Enc --> Lexer
    end
    subgraph run ["matching"]
        Router["router<br/>by pattern and measured cost"]
        Nfa["single-pass NFA<br/>no backtracking"]
        Fold["set-reachability fold<br/>balanced groups, fields, axes"]
        Gpu["CUDA scan<br/>token-kind patterns"]
        Router --> Nfa
        Router --> Fold
        Router --> Gpu
    end
    Pattern["pattern"] --> Parser["parser<br/>AST, program"] --> Router
    Lexer --> Router
    Nfa --> Out["matches<br/>spans, registers"]
    Fold --> Out
    Gpu --> Out

    style Bytes fill:#374151,stroke:#6b7280,color:#f9fafb
    style Enc fill:#1e3a8a,stroke:#3b82f6,color:#ffffff
    style Lexer fill:#1e3a8a,stroke:#3b82f6,color:#ffffff
    style Pattern fill:#374151,stroke:#6b7280,color:#f9fafb
    style Parser fill:#5b21b6,stroke:#8b5cf6,color:#ffffff
    style Router fill:#5b21b6,stroke:#8b5cf6,color:#ffffff
    style Nfa fill:#0f766e,stroke:#14b8a6,color:#ffffff
    style Fold fill:#0f766e,stroke:#14b8a6,color:#ffffff
    style Gpu fill:#9a3412,stroke:#ea580c,color:#ffffff
    style Out fill:#374151,stroke:#6b7280,color:#f9fafb

The architecture page maps every module and execution surface, and the engine explains the two matchers.

• The engine runs over the significant tokens only; whitespace never reaches it.
• A pattern the device cannot represent runs on the engine, so the two never disagree.
• Lexing across cores is byte-identical to the serial lexer, and a chunked scan returns the whole-input match set.
• On the subset shared with a regular expression, the single-pass engine reports the regex crate's spans on all 20,000 generated cases in tests/conformance.rs.

Performance

trex, the regex crate, Python's re and Perl over one generated corpus, each asked to compile, test, find, count, capture, replace and split with the same patterns. Windows 11 on a Ryzen 9 7900X, the default build with the GPU disabled, summed over the 68 rows every engine answered:

Over the 149 operations both trex and regex can express, regex is faster on 73 and trex on 76, and trex finishes the set 3.65x to 3.69x faster across two passes. The harnesses in benches/ produce every row.

Repository layout

Building and testing

cargo build --release                          # default features: gpu, tandem
cargo build --release --no-default-features    # CPU only, no CUDA dependency
cargo build --release --features compress      # adds trex compress and its 21.6 MB prior
cargo test --profile release-test              # every test target, documented examples included
cargo clippy --all-targets --all-features

The Python module builds from python/ into the active virtual environment with pip install maturin and then maturin develop --release -m python/Cargo.toml.

The PowerShell module builds with PWRS 0.3.1 from crates.io and cargo pwrs of the same release, cargo install cargo-pwrs --version 0.3.1. In powershell/, cargo pwrs build --release builds the module into target/pwrs/Trex and cargo pwrs test --release runs its Rust tests and then its Pester suites, the documented PowerShell examples among them, in PowerShell 7 and in Windows PowerShell 5.1. The report suites compare the module's output with the command's, so TREX_CLI names a trex binary built from the same tree.

Platforms and MSRV

• Rust 1.96 or later, edition 2024; the PowerShell module's crate needs 1.98.
• The command and the library are built and tested on Windows 11 x64, and the published crate builds with or without a CUDA toolkit.
• The gpu feature compiles kernels/scan.cu with nvcc at build time where nvcc is installed, and loads it through the NVIDIA driver at run time. A build without nvcc, or a machine without a device, runs on the CPU.
• Python 3.11 or later, from stable-ABI wheels for Windows x64, Linux x64 (manylinux_2_28) and macOS arm64, each tested when it is built, and from the source distribution elsewhere.
• PowerShell 7 and Windows PowerShell 5.1, from one module carrying Windows x64, Linux x64, FreeBSD x64 and macOS arm64; its Pester suites pass on each.

Wiki

The documentation site is at variably-constant.github.io/trex, built from wiki/ by .github/workflows/wiki-deploy.yml on every push to main that changes it.

It is a Hugo site using the Hextra theme, organized by the Diataxis framework: five tutorials, eight how-to guides, four explanations, and reference pages for the commands, the pattern syntax, the library API, the Python module, the PowerShell module and each property axis.

Credits and influences

• The conformance tests hold the single-pass engine to the spans of the regex crate's PikeVM, and the engine lays out a loop as that crate's compiler does.
• The set-reachability fold is the operational form of Antimirov's partial derivatives, over a token alphabet with a register environment.
• ignore, ripgrep's directory walker, applies .gitignore and .ignore rules to every directory argument.
• cudarc loads the PTX kernel and drives the device.
• Flynnel schedules the parallel lexer and the tandem CPU and GPU dispatch.
• PyO3 and maturin build the Python module.
• PWRS builds the PowerShell module.

Use of AI tools

The author used Claude (Anthropic) via the Claude Code CLI for code development assistance, documentation drafting, and benchmark scripting during the preparation of this repository. All design decisions and the final content were determined by the author. The implementation, the tests and the benchmark results were verified through zero-warning cargo clippy in four feature configurations, cargo test of every target in the default and compress builds, a test that runs every documented command against the built binary, the PowerShell module's Pester suites in PowerShell 7 and Windows PowerShell 5.1, and end-to-end runs of the release binaries on Windows 11 x64.

License

MIT, as written in LICENSE-MIT.