Crate trex 

trex: Token-Regular EXpression.

trex is a regex-shaped pattern language whose alphabet is typed tokens rather than raw bytes. A structure-aware lexer turns input into a stream of typed atoms (numbers, words, quoted strings, IPs, balanced bracket groups, punctuation), and patterns are matched over that stream by a single-pass engine, with a set-reachability engine (the operational form of the Antimirov partial derivative) for the constructs it routes away; both carry a register environment.

The result is a one-line, regex-terse surface that expresses three things regular expressions cannot:

• balanced, nestable bracket groups (the lexer pairs them, so \B(...) is a single primitive rather than an impossible recursion),
• long-distance binding (:name writes a register, =name requires an equal token later), evaluated by a register-set automaton with no catastrophic backtracking, unlike a backtracking backreference,
• content-addressed lookahead (~"lit"), answered by a presence prefilter rather than a positional scan.

Dual-grain scanning

Tokens are spans of bytes, so a scan has two stages over the same input: the byte grain lexes the bytes into tokens and the token grain matches the pattern over them. scan_dual_grain runs the two on two threads as a producer and a consumer and returns the matches scan returns. See wiki/content/docs/explanation/architecture.md for the architecture.

Re-exports

pub use context::Agreement;

pub use context::ContextConfig;

pub use context::ContextField;

pub use context::RelationContext;

pub use context::RelationReading;

pub use context::Window;

pub use context::WindowProfile;

pub use captures::CaptureSlots;

pub use captures::SlotCursor;

pub use captures::captures_read;

pub use captures::captures_read_at;

pub use captures::captures_read_iter;

pub use captures::static_captures_len;

pub use captures::static_token_extent;

pub use cursor::Cursor;

pub use cursor::MatchCursor;

pub use cursor::MatchRef;

pub use cursor::Split;

pub use cursor::capture_names;

pub use cursor::captures_at;

pub use cursor::captures_first;

pub use cursor::captures_iter;

pub use cursor::captures_len;

pub use cursor::escape;

pub use cursor::find;

pub use cursor::find_at;

pub use cursor::find_iter;

pub use cursor::is_match_at;

pub use cursor::shortest_match;

pub use cursor::shortest_match_at;

pub use cursor::split;

pub use cursor::split_at_depth;

pub use cursor::splitn;

pub use curvature::Curvature;

pub use curvature::EdgeCurvature;

pub use dual_grain::GrainTiming;

pub use dual_grain::scan_dual_grain;

pub use echo::EchoConfig;

pub use echo::EchoField;

pub use echo::EchoFrame;

pub use echo::SuperEcho;

pub use entanglement::Entanglement;

pub use custom::LibTest;

pub use custom::Precedence;

pub use custom::Rule;

pub use custom::Severity;

pub use custom::ShapeError;

pub use custom::ShapeSet;

pub use custom::TestFailure;

pub use custom::TokenShape;

pub use ast::EmptyLoop;

pub use builder::PatternBuilder;

pub use pattern_set::PatternSet;

pub use pattern_set::SetMatches;

pub use engine::INLINE_REGS;

pub use engine::Match;

pub use engine::Regs;

pub use engine::Span;

pub use engine::captures;

pub use engine::captures_over;

pub use engine::captures_over_with_lists;

pub use engine::captures_with_empty_loop;

pub use engine::captures_with_lists;

pub use engine::captures_with_shapes;

pub use engine::captures_with_shapes_and_lists;

pub use engine::is_match;

pub use engine::scan;

pub use engine::scan_with_empty_loop;

pub use engine::scan_with_shapes;

pub use parser::parse_with_empty_loop;

pub use flow::AnalyticConfig;

pub use flow::AnalyticField;

pub use flow::FlowConfig;

pub use flow::FlowField;

pub use flow::FlowFrame;

pub use flow::Signal;

pub use flow::analytic;

pub use flow::analytic_signal;

pub use gpu::Backend;

pub use gpu::BackendUsed;

pub use gpu::device_available;

pub use gpu::gpu_eligible;

pub use gpu::scan_gpu;

pub use gpu::scan_with_backend;

pub use geodesic::Geodesic;

pub use grammar::Grammar;

pub use grammar::GrammarError;

pub use grammar::Node;

pub use holography::Holography;

pub use magnitude::MagnitudeConfig;

pub use magnitude::MagnitudeField;

pub use magnitude::MagnitudeFrame;

pub use observation::ObservationConfig;

pub use observation::ObservationField;

pub use observation::ObservationFrame;

pub use orbit::OrbitGroup;

pub use orbit::OrbitToken;

pub use orbit::canonical;

pub use orbit::same_orbit;

pub use orbit::tokenize;

pub use parser::ParseError;

pub use parser::parse;

pub use parser::parse_with_inputs;

pub use typed::Clock;

pub use typed::date_order_day_first;

pub use typed::now_override;

pub use typed::set_date_order_day_first;

pub use typed::set_now;

pub use typed::set_tz_offset;

pub use profile::AxisCtx;

pub use profile::AxisProfile;

pub use profile::EchoProfile;

pub use profile::MagnitudeProfile;

pub use profile::Profile;

pub use profile::ShapeProfile;

pub use profile::SpectralProfile;

pub use profile::StressProfile;

pub use profile::fold_profiles;

pub use profile::fold_tokens;

pub use relation::Chord;

pub use relation::Edge;

pub use relation::RelationField;

pub use relation::RelationFrame;

pub use relation::RelationKind;

pub use resonator::Spectrum;

pub use resonator::analyze_symbols;

pub use resonator::over_byte_values;

pub use resonator::over_bytes;

pub use resonator::over_supertokens;

pub use resonator::over_tokens;

pub use rewrite::Field;

pub use rewrite::Keep;

pub use rewrite::Mask;

pub use rewrite::Matched;

pub use rewrite::Reference;

pub use rewrite::ReportAt;

pub use rewrite::ReportField;

pub use rewrite::ReportRule;

pub use rewrite::Spanned;

pub use rewrite::Template;

pub use rewrite::TemplateError;

pub use rewrite::redactions;

pub use rewrite::redactions_with_shapes;

pub use rewrite::rewrite;

pub use rewrite::rewrite_first;

pub use rewrite::rewrite_n;

pub use rewrite::rewrite_n_with;

pub use rewrite::rewrite_with;

pub use rewrite::rewrite_with_backend;

pub use seam::Recovery;

pub use seam::SeamConfig;

pub use seam::SeamField;

pub use shape::RegionKind;

pub use shape::ShapeConfig;

pub use shape::ShapeField;

pub use shape::ShapeFrame;

pub use spectral::CodeTexture;

pub use spectral::SpectralField;

pub use spectral::SpectralFrame;

pub use spectral::Texture;

pub use spectral::analyze;

pub use spectral::code_regions;

pub use spectral::code_texture;

pub use spectral::high_entropy_runs;

pub use spectral::regions;

pub use streaming::HeldStream;

pub use streaming::StreamScanner;

pub use streaming::scan_chunked;

pub use stress::StressConfig;

pub use stress::StressField;

pub use stress::StressFrame;

pub use supertoken::Role;

pub use supertoken::SuperToken;

pub use supertoken::supertokens;

pub use topology::Topology;

Modules

action

The action reading: the two conjugate energies trex already computes, combined.

ast

The pattern AST: the intermediate form the parser emits and the engine consumes.

bpe

Byte-pair encoding: a learned subword tokenizer (Sennrich, Haddow, Birch, ACL 2016), the data-driven token inventory for natural-language text where the rule-based lexer’s whole-token boundaries do not fit.

builder

A pattern built with its readings named, rather than parsed and taken as it comes.

byte_dfa

The determinizer over a byte-grain automaton, built as it is walked.

byte_lex

The lexer’s recognizers as byte automata.

byte_nfa

A byte-grain non-deterministic automaton, and the simulation that reads it.

byte_simd

Hand-rolled SIMD byte search: a memchr-style substring find with runtime CPU-feature dispatch (AVX2 / SSE2 / scalar) and a bit- identical contract - every path returns the same leftmost match position as the scalar baseline. This is the speed layer under the content guard, which searches a window for a literal.

bytepat

Byte-level sub-patterns: the low grain that runs inside a single token.

canon

Orbit canonicalization: the canonical representative of a symmetry orbit.

captures

A capture buffer a caller owns and refills, and the two pattern properties that are fixed before any input is seen.

context

The rolling context: every axis folded over a window that slides one unit at a time, at the token rung and the unit rung, and the agreement of the grains’ own boundaries.

cursor

Matches taken one at a time, and the operations that stop before the end.

curvature

The curvature reading - the local twist density of the relation graph.

custom

The declarations a lex and a parse share: token shapes the lexer runs alongside its built-in recognizers, token kinds declared from patterns over the stream, and named sub-patterns a \{name} inlines.

decoded

The decoded content of a token that carries an encoding: a base64 blob’s bytes and a JSON Web Token’s header and payload.

dual_grain

Dual-grain producer-consumer pipeline.

e8

The E8 root lattice and its Weyl group - the symmetry substrate under the deepest crate::orbit rung.

echo

The echo axis - trex’s recurrence substrate, the connection on the bundle.

edit

Edit distance over whole tokens: whether one text is within k edits of another, an edit being one character inserted, deleted or substituted, which is the Levenshtein distance ugrep’s -Z and spaCy’s FUZZY also count. A transposition costs two.

encoding

Input transcoding: accept text in any UTF encoding, hand the lexer UTF-8.

ends_simd

The scan under the leftmost, non-overlapping selection: the first anchor in a range whose longest match end lies past it, found a vector at a time.

engine

The matching engine.

entanglement

The entanglement reading - shared structure across a cut.

explain

What a match is made of, for --explain: the kinds of the tokens it spans, the guard each guarded kind passed to be that kind, the value of every axis the pattern read at those tokens, and the rung of the scan ladder that answered. The axes are read through the same analyses the engine builds from, once per input, so an explanation reports the number the predicate compared.

files

The inputs a command reads and the shapes it reports them in: a walk over a tree under ignore rules, the binary check, the line and column of a byte offset, and a unified diff built from the edits a rewrite makes.

flow

The flow axis - trex’s dynamics substrate (direction + rate of change).

follow

Following files as they grow: the bytes appended to each as they arrive, and a note where one was truncated, replaced under its name, or removed.

gauge

The gauge reading - the meaning left invariant when bound names change.

geodesic

The geodesic reading - distance along the relation graph, not along the line.

gpu

GPU SIMT scan backend (the default gpu feature; opt out with --no-default-features).

grammar

A universal grammar engine: a parser generator over the typed-token alphabet.

gravity

Gravity and repulsion between units: the pull the data shows one type to have on another at a gap, and three readings off it.

holography

The holography reading - the boundary determines the bulk.

index

An index over a tree that says which files a pattern cannot match, so a scan never opens them.

infer

Pattern inference from examples: the most specific pattern every example matches, read off an alignment of their token sequences. A position where every example has the same text is that literal; one where the kinds agree and the texts differ is the kind’s atom; one where the kinds differ is the class of the kinds seen; a run of one kind whose length differs between examples is that kind repeated with the bounds seen; a position some examples lack is optional. The pattern is verified against every example before it is returned.

isa

Which instruction-set rung this CPU can run, resolved once.

kind_route

A pattern that is a fixed sequence of token kinds - \W, \N, \W \N - answered over the lexer’s chunk parts in place.

lexer

The typed-atom lexer: input bytes to a Vec<Token> with bracket pairing.

library

The shipped library: named token kinds with a shape and, where the standard defines one, a checksum guard, and named sub-patterns over the stream. A \{name} reaches every entry without a declaration, and a declaration of the same name shadows it.

magnitude

The magnitude axis - trex’s scale substrate, and the energy / gradient stack that derives from it.

nfa

Single-pass NFA simulation (a Pike-style virtual machine) over the token stream.

notice

Where the notes of the scheduler trex runs its parallel work on go: a calibration it measured, a lever it read. A program embedding trex installs a sink with notice::set_sink; with none installed they print to the standard error. Where the library’s notes go.

observation

The observation axis - trex’s vantage substrate.

orbit

Orbit-based tokenization: the symmetry axis, orthogonal to frequency (BPE) and predictability (seam).

paint

Color for the reports: the depth a console renders, a color or a style sent at that depth, the roles a report paints, and the overrides --colors spells.

parallel_lex

Parallel tokenization across cores.

parser

The surface parser: a trex pattern string to a Pattern AST.

pattern_set

Many patterns asked of one input, over one lex.

prefilter

Byte-grain presence prefilter: approximate-membership filters over the n-grams of a corpus, answering “might this literal occur here?” with one-sided error and no false negatives.

profile

Axis profiles: the per-axis monoid that lifts a token reading to any unit above it.

quantity

Physical quantities: a number with a unit symbol, read as one token and compared within the unit’s family after an exact normalization to the family’s base unit.

records

The records a record-level query runs over: how an input is cut into the units a --all, --any, --none or --at-least question is asked of, each a byte span of the input. A unit is a line, a paragraph or the whole input; a run from one match of a pattern to the next, or each match of one; or a unit an axis finds with no delimiter: the stream’s own record period, the seam axis’s segments, the spectral texture regions, the shape axis’s regions, or a supertoken.

relation

The relation tier - trex’s two-point substrate.

report

The text a scan’s report is made of, for every surface that writes one: a match as a JSON object, its registers as JSON with each typed value beside its text, an explanation as JSON, a match as the one-input report writes it, and a line of an input with the matches in it painted; and the reports built from them, each line handed to a sink the surface supplies, so the command line and every module write one report the same way.

resonator

A resonator bank: the filterbank with complex poles, so a band has a frequency and not only a decay.

rewrite

Rewrite and transform: match, render a replacement template per match, splice the renders into the input.

rule_scan

The rules of pattern files scanned over an input: the rules that fire on each match as one set per list of files they read, and the rules that fire on records each as a record query, every finding reported under its rule with the message rendered from the match, its severity and its fix.

seam

Seam: bidirectional predictive segmentation.

shape

The shape axis - trex’s structural substrate, the dual of spectral.

spectral

The spectral axis: trex’s temporal substrate.

streaming

Streaming scan: feed the input in chunks and recover exactly the matches a whole-input scan would produce.

stress

The stress axis - trex’s structural-load substrate.

supertoken

The supertoken tower: a grammar-free unit one level above the token.

tandem

CPU + GPU tandem dispatch (the default tandem feature, which implies gpu).

templates

Log-template mining: the lines of an input grouped by their token-kind silhouette, each group read as one template whose positions are the literal text every line shares or a slot named by the kind that varies there, with the lines each template covers; and the rarity of a line’s template, which @shape:rare reads at every token of the line.

token

The typed-token contract shared by the lexer and the engine.

tokutil

Substrate token utilities shared across the field axes.

topology

The topology reading - the global loop structure of the relation graph.

trace

Which rung of a ladder answered a call.

typed

Typed value predicates: a kind atom’s {...} body compared in the type’s own units.

window

The part of an input a head, a tail or a line range selects, and the readers that fetch only that much of it.

Functions

version

The crate version string, sourced from the package manifest.