# tokmat
[](https://github.com/Jrakru/tokmat/actions/workflows/ci.yml)
[](https://docs.rs/tokmat)
[](https://crates.io/crates/tokmat)
`tokmat` is a standalone Rust crate for metadata-driven tokenization and TEL-based extraction of
Canadian-style address strings.
It is the low-level parsing core: other crates can build strategies, pipelines, analytics, or
language bindings on top of it without pulling in broader workspace assumptions.
`tokmat` now uses PCRE2 as its runtime regex engine across tokenization, TEL compilation, and
extractor execution.
## Highlights
- Standalone core crate with no sibling-workspace runtime assumptions
- PCRE2-only runtime regex path across tokenization and extraction
- Metadata-driven TEL extraction over token classes instead of raw-text-only matching
- File-backed token models plus inline/in-memory model support
- Per-model, configurable word definition (the character class that delimits
tokens) — loaded from `WORDDEFINITION.param` and applied per model for
tokenization, with no process-global side effects from loading a model
- Reference corpus tests, doctests, linting, and publish dry-run validation
## Why this crate exists
`tokmat` separates address parsing into two explicit phases:
1. Tokenization and classification
2. TEL-driven extraction over token classes
That split keeps the parser predictable.
- Tokenization decides where boundaries are.
- Classification decides what each token is.
- TEL decides which token-class sequence to match and what to capture.
This is a better fit for messy address data than pushing everything into one monolithic regex.
## Parsing model
```text
Raw input
|
v
+---------------------------+
|
v
+---------------------------+
| tokenize into boundaries |
| ex: ["123", " ", "MAIN"] |
+---------------------------+
|
v
+---------------------------+
| classify each token |
| ex: ["NUM", " ", "ALPHA"] |
+---------------------------+
|
v
+---------------------------+
| compile TEL pattern |
| ex: <<NUM#>> <<NAME@+>> |
+---------------------------+
|
v
+---------------------------+
| match on class stream |
| capture named fields |
+---------------------------+
```
The important design point is that TEL operates over token metadata, not only raw characters.
## Extractor entry modes
The extractor exposes two ways to run TEL:
- `parse_tokens(...)`
- `compile_pattern(...)` + `parse_compiled_tokens(...)`
They are not two different extractors. They are two entry points into the same extractor runtime.
```text
Compat path
pattern string
-> compile or fetch compiled TEL pattern
-> build/fetch object plan
-> run extractor
Precompiled path
compiled pattern
-> build/fetch object plan
-> run extractor
```
### When to use each
Use `parse_tokens(...)` when:
- you want the simplest API
- patterns are dynamic or user-supplied
- you are fine relying on the internal compiled-pattern cache
Use `compile_pattern(...)` + `parse_compiled_tokens(...)` when:
- you load a fixed TEL set once and reuse it many times
- you want TEL validation to happen up front
- you expect high pattern churn or a tiny compiled-pattern cache
### Which API should I call?
Use this rule of thumb:
```text
Do you already have a compiled TEL set that will be reused?
|
+-- no -> use parse_tokens(...)
|
+-- yes -> use parse_compiled_tokens(...)
```
Another way to say it:
- application code and ad hoc parsing usually want `parse_tokens(...)`
- long-lived workers, services, and batch pipelines usually want precompiled TEL patterns
### Why they can benchmark the same
On the reference corpus used by this crate:
- `695` extractor cases
- `344` unique TEL patterns
- default compiled-pattern cache capacity: `512`
That means the compat path quickly warms the cache and then behaves almost like the precompiled
path. In the 10MM volume benchmark the two extractor modes were effectively identical:
```text
10MM operations, default cache sizes
extractor-compat 30,407 ops/s 16.8 MB RSS
extractor-precompiled 30,127 ops/s 16.2 MB RSS
```
That result does not mean precompiled mode is useless. It means the current corpus is
cache-friendly.
### When precompiled actually matters
Under cache pressure, precompiled mode separates clearly. With the compiled-pattern cache forced to
capacity `1`:
```text
1MM operations, compiled-pattern cache = 1
extractor-compat 12,828 ops/s 7.2 MB RSS
extractor-precompiled 30,609 ops/s 12.5 MB RSS
precompiled vs compat: 2.386x faster
```
Interpretation:
- `compat` is the convenience API
- `precompiled` is the explicit reuse API
- on cache-friendly workloads they converge
- on churn-heavy workloads precompiled mode avoids repeated TEL compilation cost
## TEL in one page
TEL stands for Token Extraction Language.
A TEL pattern is made of typed segments:
- Captures: `<<FIELD>>`
- Captures with type modifiers: `<<STREET@+>>`
- Explicit class constraints: `<<TYPE::STREETTYPE>>`
- Vanishing groups: `<!PROV!>`, `<!@!>`, `<!DROP@!>`, `<!DROP::STREETTYPE!>`, or `<!,!>`
- Optional vanishing groups: `<!DASH?!>`, `<!@%?!>`, `<!DROP@%?!>`, `<!,?!>`, or `<!{{#}}?!>`
- Literal blocks: `{{PO BOX}}`
Common modifiers:
- `@` alpha-like token matching
- `#` numeric token matching
- `%` extended token matching
- `+` one or more
- `?` optional
- `$` greedy matching
- `::CLASSNAME` explicit class assignment
Examples:
- `<<CIVIC#>> <<STREET@+>> <<TYPE::STREETTYPE>>`
- `{{PO BOX}} <<BOXNUM#>>`
- `<<CITY@+$>> <<PROV::PROV>> <<PC::PCODE>>`
- `<<UNIT::NUM>> <!DASH!> <<CIVIC::NUM>> <<STREET::NUM>> <<TYPE::STREETTYPE>>`
The last example intentionally requires a street number after the dashed civic
range. It accepts shapes such as `11-47 7 ST`, but does not accept `11-47 ST`
where it is unclear whether the final number is missing or the civic range was
misread as part of the street.
See [`docs/TEL_SPEC.md`](docs/TEL_SPEC.md) for a cleaner language reference.
## Quick start
### In-memory token model
This example keeps the model inline so it is easy to understand and compiles without external
files.
```rust
use std::collections::HashSet;
use tokmat::extractor::Extractor;
use tokmat::tokenizer::{tokenize_and_classify, TokenClassList, TokenDefinition};
let token_definitions: TokenDefinition = vec![
("NUM".into(), r"\d+".into()),
("ALPHA".into(), r"[A-Z]+".into()),
("ALPHA_EXTENDED".into(), r"[A-Z][A-Z'\\-]*".into()),
];
let token_class_list: TokenClassList = vec![
("STREETTYPE".into(), HashSet::from(["ST".to_string(), "AVE".to_string()])),
];
let tokenized = tokenize_and_classify(
"123 MAIN ST",
&token_definitions,
Some(&token_class_list),
);
assert_eq!(tokenized.tokens, vec!["123", " ", "MAIN", " ", "ST"]);
assert_eq!(tokenized.types[0], "NUM");
let extractor = Extractor::new(token_definitions, token_class_list);
let (_, fields, complement) =
extractor.parse_string("123 MAIN ST", "<<CIVIC#>> <<NAME@+>> <<TYPE::STREETTYPE>>")?;
assert_eq!(fields.get("CIVIC").map(String::as_str), Some("123"));
assert_eq!(fields.get("NAME").map(String::as_str), Some("MAIN"));
assert_eq!(fields.get("TYPE").map(String::as_str), Some("ST"));
assert_eq!(complement, "");
# Ok::<(), tokmat::error::ParseError>(())
```
### File-backed token model
If you already have a model directory in the wanParser-style layout:
```text
model/
TOKENDEFINITION/TOKENDEFINITONS.param2
TOKENCLASS/*.param
```
you can load it directly:
```rust,no_run
use tokmat::extractor::Extractor;
use tokmat::token_model::TokenModel;
use tokmat::tokenizer::tokenize_with_model;
let model = TokenModel::load("tests/fixtures/model_1")?;
let tokenized = tokenize_with_model("123 MAIN ST", &model);
let extractor = Extractor::new(
model.token_definitions().clone(),
model.token_class_list().clone(),
);
let (_, fields, _) =
extractor.parse_string("123 MAIN ST", "<<CIVIC#>> <<NAME@+>> <<TYPE::STREETTYPE>>")?;
assert_eq!(tokenized.tokens[0], "123");
assert_eq!(fields.get("CIVIC").map(String::as_str), Some("123"));
# Ok::<(), Box<dyn std::error::Error>>(())
```
### Word definition (per model)
The "word definition" is the character class that decides where tokens break.
It defaults to `\w\-'` (word chars, hyphen, apostrophe) and can be overridden per
model with an optional `WORDDEFINITION.param` file in the model directory:
```text
model/
TOKENDEFINITION/TOKENDEFINITONS.param2
TOKENCLASS/*.param
WORDDEFINITION.param # first non-comment line is the char class, e.g. [\w\-'.]
```
The definition is **per model**: a `TokenModel` carries its own `WordDefinition`
and compiled boundary, and tokenization uses that boundary directly. Loading a
model has **no process-global side effect**, so several models with different
word definitions can coexist in one process without contaminating each other.
```rust,no_run
use tokmat::token_model::TokenModel;
use tokmat::tokenizer::{split_input_tokens_with, tokenize_with_model};
let model = TokenModel::load("tests/fixtures/model_1")?;
// Inspect the active definition (char class, e.g. `\w\-'`).
let _chars = model.word_definition().chars();
// Tokenize using this model's boundary (no global state):
let tokens = split_input_tokens_with("192.168 MAIN", model.word_boundary());
// With the default definition, `.` is a boundary -> "192" / "168" split.
// A model whose WORDDEFINITION.param is `[\w\-'.]` keeps "192.168" as one token.
let _ = (tokens, tokenize_with_model("192.168 MAIN", &model));
# Ok::<(), Box<dyn std::error::Error>>(())
```
That model-owned boundary is important when you write TEL for punctuation. TEL
matches the **class stream** created by the model, then returns aligned raw text
for captures:
| Hyphen is word-like | `NUM_EXTENDED ALPHA STREETTYPE` | `<<CIVIC::NUM_EXTENDED>> ...` |
| Hyphen is a literal boundary | `NUM - NUM ALPHA STREETTYPE` | `<<UNIT::NUM>> <!{{-}}!> <<CIVIC::NUM>> ...` |
| Hyphen is classified | `NUM DASH NUM ALPHA STREETTYPE` | `<<UNIT::NUM>> <!DASH!> <<CIVIC::NUM>> ...` |
Use `<!DASH?!>` when the stream contains a `DASH` class. Use `<!-?!>` when the
stream contains a literal `-`, but note that the default word definition absorbs
hyphen into extended classes such as `NUM_EXTENDED`, `ALPHA_EXTENDED`, and
`ALPHA_NUM_EXTENDED`, so there is often no standalone `-` token to vanish.
Direct literal punctuation such as `<!,?!>` is allowed inside a vanishing group.
Escape punctuation that is also TEL syntax with a literal block, for example
`<!,{{?}}?!>` for an optional literal `,?` sequence. Type modifiers also work
inside vanishing groups: `<!@!>` consumes one alpha token without capturing it,
`<!DROP@!>` does the same with a readability label that is discarded, and
`<!DROP::STREETTYPE!>` consumes a street-type class without adding `DROP` to the
output. `<!{{@}}!>` consumes a literal `@` token. A literal block such as
`<!{{1-1}}?!>` matches that exact class-stream text; it does not match `NUM DASH
NUM` or a single extended token after the model has classified it.
## Two-phase extraction
The crate is easiest to reason about when you think in phases.
### Phase 1: tokenization
Input:
```text
APT-210 O'CONNOR ST
```
Boundary handling preserves address-relevant shapes:
```text
["APT-210", " ", "O'CONNOR", " ", "ST"]
```
This matters because `APT-210` and `O'CONNOR` should not be destroyed by a simplistic
whitespace-only split.
### Phase 2: metadata-driven extraction
Once each token has a type or class, TEL matches over the class sequence rather than blindly over
raw characters.
Example:
```text
Tokens : ["123", " ", "MAIN", " ", "ST"]
Types : ["NUM", " ", "ALPHA", " ", "ALPHA"]
Class : ["NUM", " ", "ALPHA", " ", "STREETTYPE"]
TEL : <<CIVIC#>> <<NAME@+>> <<TYPE::STREETTYPE>>
```
The `TYPE` field is extracted because `ST` is known to belong to the `STREETTYPE` class.
That is the metadata-driven part of the design: the extraction rule is not just matching the text
`"ST"`, it is matching the semantic class attached to that token.
## Benchmarks
### 0.3.0 performance work (1M-row benchmark via `tokmat-polars`)
Measured on a 1M-row Canadian-address corpus through the `tokmat-polars`
criterion harness (wall-clock for 1,000,000 rows). These reflect `tokmat`-core
changes in 0.3.0 — the single-pass classifier, atomic counters, and especially
removing a per-row PCRE2 JIT-stack `mmap` (the dominant extraction cost):
```text
1,000,000 rows 0.2.x 0.3.0 speedup
tokenize 5.22 s 0.66 s ~8x
extract (from pre-tokenized input) ~7.4 s ~1.0 s ~7x
extract (from raw strings) ~12.5 s 3.80 s ~3.3x
```
Takeaway:
- The per-row JIT-stack `mmap` (from `captures()` allocating a fresh `MatchData`
with a 5 MB JIT stack) dominated extraction and serialized parallel work on the
kernel `mmap` lock; removing it is the single biggest win.
- Tokenization also gained from passing each model's boundary explicitly instead
of reading a process-global lock per row.
### Earlier benchmark snapshots
The benchmark scripts and JSON artifacts used during crate extraction live in the parent repo:
- `scripts/benchmark_tokmat_variants.py`
- `scripts/benchmark_extractor_mode_tradeoffs.py`
Two benchmark snapshots are especially useful:
### PCRE2-only crate vs earlier mixed-engine crate
```text
10MM operations
tokenizer
mixed engines : 354,382 ops/s 6.1 MB RSS
pcre2 only : 564,171 ops/s 3.6 MB RSS
extractor-compat
mixed engines : 30,407 ops/s 16.8 MB RSS
pcre2 only : 30,435 ops/s 12.6 MB RSS
extractor-precompiled
mixed engines : 30,127 ops/s 16.2 MB RSS
pcre2 only : 30,168 ops/s 12.6 MB RSS
```
Takeaway:
- PCRE2-only materially improves tokenizer throughput
- extractor throughput stays essentially flat
- RSS drops across the measured workloads
### Extractor mode trade-off under cache pressure
```text
1MM operations, compiled-pattern cache = 1
extractor-compat 12,828 ops/s 7.2 MB RSS
extractor-precompiled 30,609 ops/s 12.5 MB RSS
```
Takeaway:
- default corpus + default cache sizes make compat and precompiled look similar
- precompiled mode matters when many pattern compiles would otherwise be repeated
- if you do not know yet, start with `parse_tokens(...)` and only move to precompiled patterns
when you need explicit reuse or validation
## What makes the crate polished for publication
- Standalone fixture corpus under `tests/`
- Strict linting through Clippy
- Complexity gate validated during development
- Formal TEL grammar in `grammar/tel.ebnf`
- Public docs suitable for crates.io and docs.rs
## Release workflow
`tokmat` can be published from GitHub Actions on tag pushes that match `v*`.
The CI workflow already validates formatting, Clippy, tests, docs, and a
publish dry-run. The release workflow should remain limited to crates.io
publication because this repository is the parser kernel, not the Python/Polars
distribution surface.
Release steps:
1. Update the version in `Cargo.toml` under `[package] version` to `0.2.0`.
2. Commit the version bump.
3. Create and push a tag matching the version:
```bash
VERSION=0.2.0
git add -A
git commit -m "Release ${VERSION}"
git tag "v${VERSION}"
git push origin "v${VERSION}"
```
Before the first release, add a crates.io API token to the repository secrets
as `CARGO_REGISTRY_TOKEN`.
## Limitations
- The crate is intentionally low-level. It does not try to solve full multi-strategy address
interpretation by itself.
- TEL is powerful, but it assumes you have a reasonable token model.
- The API focuses on extraction primitives; higher-level strategy orchestration belongs in layers
above this crate.
## Security: PCRE2 linkage
`tokmat` runs all regex through PCRE2. Because token-definition and TEL patterns
are compiled and matched by PCRE2, the regex engine version matters:
- **CVE-2025-58050** is a heap-buffer-overflow read in PCRE2's `(*scs:...)` /
`(*ACCEPT)` "scan substring" path. It affects **PCRE2 10.45** and is fixed in
**10.46**. It is only reachable via attacker-controlled *regex* (a malicious
token-definition or TEL pattern), not via parsed input strings.
- `pcre2-sys` vendors a patched PCRE2 **10.46**, but by default it prefers a
system `libpcre2-8` if `pkg-config` finds one — so a host with system PCRE2
10.45 would be exposed. This repo ships a `.cargo/config.toml` that sets
`PCRE2_SYS_STATIC=1`, statically linking the patched vendored copy so the
engine version is deterministic.
- **If you depend on `tokmat`, set `PCRE2_SYS_STATIC=1` in your own build** (or
ensure your system PCRE2 is >= 10.46). Treat token-definition / TEL patterns as
trusted configuration, not untrusted input.
## License
MIT. See the `LICENSE` file in the crate root.