precursor-0.2.3 is not a library.

precursor

precursor is a CLI for pre-protocol payload tagging + similarity clustering across packet payloads, logs, and raw binary fragments. It combines PCRE2 named-capture matching, TLSH/LZJD/FBHash similarity, optional MRSHv2 adapter mode, and JSON outputs designed for fast triage loops before full parser or protocol tooling exists.

Release 0.2.1 Highlights

Area	What landed
Packet Inference	Single-packet protocol scoring via `-P` / `-A` / `-k`
Blob + Binary Processing	`-z, --input-blob` and `-B, --input-binary` for multiline and raw-byte stream analysis
Similarity Workflows	TLSH, LZJD, or FBHash clustering + protocol hints (`--protocol-hints`) for discovery loops
Sigma Compatibility	`--sigma-rule` converts selectors into named PCRE captures and enforces Sigma `condition` logic
Regex Engine Scaffold	`--regex-engine pcre2
Output Contract	Stable `protocol_*`, `similarity_hash`, `tags`, `xxh3_64_sum` JSON fields
Reliability	Runtime ingest path no longer relies on panic-prone `expect(...)` calls
Scenario Corpus	Versioned packet/firmware/ICS + public PCAP/log/Sigma-derived samples in `samples/scenarios/`
Release Ops	Dependency auto-bump/tag workflows + benchmark harness + Pages site

60-second teaser

cat samples/scenarios/pre-protocol-packet-triage/payloads.b64 \
  | precursor -p samples/scenarios/pre-protocol-packet-triage/patterns.pcre \
      -m base64 -t -d --similarity-mode lzjd -P --protocol-hints

Representative output shape:

{
  "tags": ["http_method"],
  "similarity_hash": "lzjd:128:...",
  "protocol_label": "http",
  "protocol_confidence": 0.93,
  "protocol_candidates": [
    {"protocol": "http", "score": 0.93, "evidence": ["matched HTTP request/headers"]}
  ],
  "xxh3_64_sum": "..."
}

Why this matters:

Fast pre-protocol triage when DPI/parsers are unavailable.
Stable JSON fields for SOC pipelines and enrichment tooling.
Built-in cluster context for LLM-assisted protocol discovery.

Best Fit

Rapid triage of payload lines from logs, brokers, sensors, or ad-hoc captures.
Label-first workflows where regex capture names become downstream tags.
Similarity-first clustering when full protocol parsers are unavailable or too brittle.
Early-stage protocol discovery where hints are fed to humans or LLM tooling.

Non-Goals

Not a replacement for full IDS/NSM stacks (Suricata, Zeek).
Not a malware rule engine replacement (YARA / YARA-X).
Not a full protocol parser stack; this is pre-protocol triage and clustering.

Architecture

Install

Cargo

cargo install precursor

From source

git clone https://github.com/Obsecurus/precursor.git
cd precursor
cargo build --release
./target/release/precursor --help

Release archives include generated shell completion files (bash, fish, zsh, powershell) and a precursor.1 man page.

Optional: Build with MRSHv2 adapter mode

mock_dir="$(mktemp -d)"
ci/build_mrshv2_mock.sh "$mock_dir"
PRECURSOR_MRSHV2_LIB_DIR="$mock_dir" cargo build --features similarity-mrshv2

Quick start

1) Match string payloads from stdin

printf 'hello world\nbye world\n' \
  | precursor '(?<hello_tag>hello)' -m string

2) Match base64 payloads

printf 'aGVsbG8gd29ybGQ=\n' \
  | precursor '(?<hello_tag>hello)' -m base64

3) Load patterns from file

printf 'aGVsbG8gd29ybGQ=\n' \
  | precursor -p patterns/new -m base64

4) Extract payload from JSON before matching

printf '{"payload":"aGVsbG8gd29ybGQ="}\n' \
  | precursor '(?<hello_tag>hello)' -j '.payload' -m base64

5) Enable similarity diffing (TLSH default)

cat payloads.b64 \
  | precursor -p patterns/new -m base64 -t -d -x 80

6) Switch to LZJD similarity mode

cat payloads.raw \
  | precursor -p patterns/new -m string -t -d --similarity-mode lzjd -x 80

7) Switch to FBHash similarity mode

cat payloads.raw \
  | precursor -p patterns/new -m string -t -d --similarity-mode fbhash -x 90

8) Emit protocol-discovery hints for an LLM loop

cat payloads.b64 \
  | precursor -p patterns/new -m base64 -t -d --protocol-hints --protocol-hints-limit 20

9) Enable single-packet protocol inference output

cat payloads.b64 \
  | precursor -p patterns/new -m base64 -P -A 0.7 -k 5

10) Match a multiline payload as one blob

printf 'GET /blob HTTP/1.1\nHost: blob.example\n' \
  | precursor '(?<multi>GET /blob HTTP/1\.1\nHost: blob\.example)' -m string -z

11) Match a raw-binary blob (short flag)

printf '\x7fELF\x02\x01\x01\x00' \
  | precursor '(?<elf_magic>^\x7fELF)' -B

12) Load Sigma rules with condition gating

cat samples/scenarios/sigma-linux-shell-command-triage/payloads.log \
  | precursor --sigma-rule samples/scenarios/sigma-linux-shell-command-triage/sigma_rule.yml \
      -m string -t -d --similarity-mode lzjd

13) Run vectorscan compatibility mode (PCRE2 fallback)

cat payloads.raw \
  | precursor '(?<http_get>GET)' -m string --regex-engine vectorscan

14) Replay a real public Log4Shell PCAP (FBHash mode)

cat samples/scenarios/public-log4shell-foxit-pcap/payloads.string \
  | precursor -p samples/scenarios/public-log4shell-foxit-pcap/patterns.pcre \
      -m string -t -d --similarity-mode fbhash -P --protocol-hints

15) Triage public firmware blobs in binary folder mode

precursor -p samples/scenarios/public-firmware-binwalk-magic/patterns.pcre \
  -f samples/scenarios/public-firmware-binwalk-magic/blobs \
  --input-mode binary -t -d --similarity-mode lzjd -P --protocol-hints

CLI reference

precursor [PATTERN] [OPTIONS]

At least one pattern source is required: positional PATTERN, --pattern-file, or --sigma-rule.

Pattern source:

positional PATTERN (single named-capture regex)
-p, --pattern-file <PATH> (one named-capture pattern per line)
--sigma-rule <PATH> (Sigma YAML selectors converted to named-capture PCRE patterns with condition enforcement)

Input:

-f, --input-folder <PATH>: read newline-delimited content from files
stdin: read newline-delimited input from standard input
-z, --input-blob: process each input source as one blob instead of line splitting
-B, --input-binary: treat each source as raw binary bytes (implies blob processing semantics)
-m, --input-mode <base64|string|hex|binary>: decode mode (default: base64)
-j, --input-json-key <QUERY>: extract payload from JSON input first

Similarity:

-t, --tlsh: compute TLSH hash for matched payloads
-d, --tlsh-diff: compute pairwise TLSH distance among matched payloads
-a, --tlsh-algorithm <48_1|128_1|128_3|256_1|256_3>
-x, --tlsh-distance <N>: max distance threshold (default: 100)
-l, --tlsh-length: include payload length in diff scoring
-y, --tlsh-sim-only: only output payloads that have TLSH similarities
--similarity-mode <tlsh|lzjd|mrshv2|fbhash>:
- tlsh, lzjd, and fbhash are implemented in default builds
- mrshv2 is implemented behind --features similarity-mrshv2 and native adapter linking
- fbhash currently uses an in-tree FBHash-inspired chunk-vector model for stream-friendly pairwise scoring
--protocol-hints: emit LLM-oriented protocol-discovery hint JSON to stderr
--protocol-hints-limit <N>: limit hint candidate count (default: 25)
-P, --single-packet: enable heuristic protocol inference on each matched payload
-A, --abstain-threshold <0.0-1.0>: minimum confidence required to emit a non-unknown label (default: 0.65)
-k, --protocol-top-k <N>: candidate count included in protocol_candidates (default: 3)
--regex-engine <pcre2|vectorscan>: regex engine selection (vectorscan mode emits compatibility checks and executes through current PCRE2 path)

Other:

-s, --stats: emit run statistics JSON to stderr

Output model

Each matched payload is emitted as JSON on stdout with fields such as:

tags: array of matched capture names
tlsh: active similarity hash when enabled (legacy field name preserved for compatibility)
similarity_hash: backend-agnostic similarity hash field
xxh3_64_sum: stable payload key for report correlation
tlsh_similarities: distance map when --tlsh-diff is enabled
protocol_label: top protocol guess (or unknown when abstaining)
protocol_confidence: confidence score for protocol_label
protocol_abstained: whether inference abstained under threshold
protocol_candidates: scored candidate list with evidence strings
sigma_rule_matches: Sigma rule titles whose condition evaluated true (when --sigma-rule is used)
sigma_rule_ids: stable Sigma rule IDs/slugs that evaluated true

When --stats is enabled, a summary JSON object is emitted to stderr. See STATS.md for schema, field meanings, and jq examples. When --protocol-hints is enabled, an additional hint JSON block is emitted to stderr for LLM-guided protocol discovery workflows, including protocol_* fields when single-packet inference is enabled. When both --single-packet and --tlsh-diff are enabled, protocol confidence is cluster-boosted using similarity neighbor counts. When --input-blob is enabled (or --input-binary is set), each file/stdin stream is treated as a single candidate payload.

Stats quick view

cat payloads.b64 \
  | precursor -p patterns/new -m base64 -t -d --similarity-mode lzjd --stats \
  1>/tmp/records.ndjson 2>/tmp/stats.json

jq '.Environment + {input_count: .Input.Count, similarities: .Compare.Similarities}' /tmp/stats.json

Notes:

Compare may be empty when too few matched payloads produce pairwise distances.
Historical record field names like tlsh_similarities are preserved for compatibility across TLSH/LZJD/FBHash modes.

Positioning vs adjacent tools

Use Suricata/Zeek for full protocol-aware IDS/NSM and rich ecosystem integrations.
Use YARA/YARA-X for signature-based scanning of files and malware-centric workflows.
Use Sigma for backend-agnostic detection content and SIEM portability.
Use Precursor when you need lightweight payload tagging + similarity clustering, or when you want to run Sigma keyword intent directly against raw payload streams via --sigma-rule.

Scenario corpus and demos

Scenario corpus: samples/scenarios/
Demo runner: samples/scenarios/run_all.sh
Static demo site source: site/
Includes packet/firmware/ICS plus public PCAP-derived Log4Shell probes, real fox-it Log4Shell PCAP replay, Sigma shell-command triage, Zeek DNS log triage, and real binwalk firmware blob samples.
Site includes mini replay reels that visually walk through PCAP replay, firmware blob triage, and Sigma labeling behavior.
tshark is only required when regenerating PCAP-derived payload extracts.

samples/scenarios/run_all.sh ./target/release/precursor

Benchmarks

Generate a reproducible scenario snapshot:

cargo build --release
ci/benchmark_scenarios.sh ./target/release/precursor benchmarks/latest.md

Committed baseline:

benchmarks/baseline-2026-02-13.md

GitHub Pages + precursor.hashdb.io

Pages workflow: .github/workflows/pages.yml
Site content: site/
Custom domain file: site/CNAME
Configure DNS at your provider with:
- record type: CNAME
- host: precursor
- value: obsecurus.github.io

Current roadmap

See ROADMAP.md for prioritized milestones and release criteria. See SIMILARITY_BACKENDS.md for MRSHv2/FBHash feasibility and backend sequencing. See SIGMA_INTEGRATION.md for Sigma feature coverage and next steps. See HARDWARE_ACCELERATION.md for regex acceleration/offload strategy. See STATS.md for run-statistics schema and usage guidance.

Development

cargo fmt --all --check
cargo test --workspace
mock_dir="$(mktemp -d)"
ci/build_mrshv2_mock.sh "$mock_dir" >/dev/null
PRECURSOR_MRSHV2_LIB_DIR="$mock_dir" LD_LIBRARY_PATH="$mock_dir:${LD_LIBRARY_PATH:-}" cargo test --workspace --features similarity-mrshv2

CI currently tests multiple toolchains and targets, including a pinned Rust 1.86.0 lane. Dependabot is configured for weekly Cargo and GitHub Actions updates, with auto patch-version bump + auto-tag workflows so dependency updates can flow into release builds.

Background

GreyNoise blog: https://www.greynoise.io/blog/precursor-a-quantum-leap-in-arbitrary-payload-similarity-analysis
GreyNoise Labs writeup: https://www.labs.greynoise.io/grimoire/2023-10-11-precursor/

License

Dual-licensed under MIT and Unlicense.

precursor 0.2.3