dupehell 0.5.1

Synthetic data generator for record linkage benchmarking.
Rust + Python — 40 domains, 500K+ rec/s.

Generate synthetic multi-entity datasets with realistic schemas, controlled duplicates, hard negatives, and ground-truth labels. Designed for benchmarking entity resolution (deduplication) and record linkage pipelines.

Quick start

Python (pip)

pip install dupehell

from dupehell import generate

r = generate(domain="publishing", size=10000, seed=42, difficulty="hell")
print(r.dataset)       # ./publishing_<hash>.parquet
print(r.ground_truth)  # ./publishing_<hash>_ground_truth.parquet
print(r.total_records) # ~10150

CLI (Rust)

cargo run --release -- --domain kyc --size 100000 --seed 42

Output

Each run produces:

• {domain}_{hash}.parquet — main dataset
• {domain}_{hash}_ground_truth.parquet — ground-truth labels
• {domain}_{hash}_nodes.parquet / {domain}_{hash}_edges.parquet — property graph (only with --graph / generate_graph=True, see below)

CLI options

Graph generation

Add --graph (CLI) or generate_graph=True (Python) to additionally emit a property graph alongside the usual tabular dataset — nodes (one per record, same attributes as the dataset) and typed edges (fk, exact_dup, hard_neg) linking record_ids that a record-linkage/graph pipeline should resolve to the same entity or a hard negative. Disabled by default: tabular output, RNG sequence, and memory footprint are unchanged when omitted.

r = generate(domain="fintech", size=10000, seed=42, generate_graph=True)
print(r.nodes)  # ./fintech_<hash>_nodes.parquet
print(r.edges)  # ./fintech_<hash>_edges.parquet

cargo run --release -- --domain fintech --size 10000 --seed 42 --graph

Features

• 40 domains — KYC, publishing, fintech, blockchain, technology, banking, healthcare, ecommerce, automotive, cybersecurity, gaming, and 30 more
• Multi-entity schemas — 3–5 entity types per domain (person, account, address, transaction)
• Controlled noise — typos, OCR errors, homoglyphs, date swaps, phonetic variants, Unicode pollution
• Hard negatives — same_field, mix_identifier, mix_arrays primitives
• Ground truth — full match labels (exact_dup, hard_neg, singleton) with cluster statistics
• Graph generation — optional property graph output (nodes, typed edges) for graph-based entity resolution and community detection benchmarking
• Deterministic — seeded RNG (rand_pcg) for reproducible output

Performance

All runs on Lenovo ThinkPad P16 Gen 2 — Intel Core i7 13th, 32 GB DDR5, SK Hynix 1 TB NVMe. Difficulty hell, IPC format. Throughput averaged across all 40 domains.

Multi-domain throughput (hell, IPC)

¹ Refreshed 2026-07-24 against the current pipeline (post multi-pass hell tier, see docs/BENCHMARK.md); 5M/10M/20M rows are from the prior sweep, not yet rerun. kyc is a persistent outlier — its natural_person entity has 23 columns (2-3x most domains), which disproportionately multiplies the cost of hell's 3 noise passes. Excluding kyc, the 1M average is 152,535 rec/s.

IPC vs Parquet

Difficulty hell, domain-average throughput.

Not yet refreshed against the current pipeline (see footnote above) — kept as the last measured baseline.

See docs/BENCHMARK.md for the full per-domain matrix (all 3 difficulties, --graph overhead, FK edge weighting) and single-domain metrics at scale.

Architecture

lib.rs / main.rs → Context (133 pools) → PipelineConfig → run_pipeline()
                                                          │
         ┌────────────────────────────────────────────────┼────────────────────┐
         ▼                                                ▼                    ▼
  entity_gen.rs                                    fk_remap.rs           hn_common.rs
  (batch gen)                                      (FK cross-ref)        (hard negatives)
         │                                                │                    │
         └────────────────────────────────────────────────┴────────────────────┘
                                                          ▼
                                                     pipeline.rs
                                               (merge + GT + IPC write)
                                                          ▼
                                               {domain}.ipc + GT.ipc

Documentation

Domains

Academia · Agriculture · Automotive · Aviation · Banking · Biotech · Blockchain · Construction · CRM · Cybersecurity · Ecommerce · Education · Energy · Fashion · Fintech · Food & Beverage · Gaming · Healthcare · Hospitality · HR · Insurance · KYC · Legal · Logistics · Manufacturing · Maritime · Media · Mining · Nonprofit · Pharma · Publishing · Real Estate · Renewable Energy · Retail · Social Media · Sports · Supply Chain · Technology · Telecom · Travel

Development

cargo test        # 113 tests, ~30s
cargo build --release
cargo clippy      # 0 warnings
cargo fmt --check # all formatted

Python wheel

pip install maturin
maturin build --release
pip install target/wheels/dupehell-*.whl

License

MIT — Educational Use Only.

This software generates synthetic data for research and educational purposes only. It must not be used for fraud, identity theft, surveillance, or any illegal activity. See ETHICS.md for the full list of prohibited uses and responsible disclosure policy.

If you use DupeHell in your research, please cite:

@software{dupehell2026,
  author = {DupeHell Contributors},
  title = {DupeHell: Synthetic Multi-Domain Dataset Generator for
           Record Linkage Benchmarking},
  year = {2026},
  url = {https://github.com/vntoinekaio/DupeHell}
}