# Getting Started
## Installation
### Python (pip)
```bash
pip install dupehell
```
### Rust (from source)
```bash
git clone https://github.com/vntoinekaio/DupeHell
cd dupehell
cargo build --release
./target/release/dupehell --domain kyc --size 1000 --seed 42
```
---
## Quick start
### Python
```python
from dupehell import generate
r = generate(
domain="publishing", size=10000, seed=42, difficulty="hell",
output_dir="./data", pools_dir="./assets/pools", schemas_dir="./schemas",
)
print(r.dataset) # ./data/publishing_<hash>.parquet
print(r.ground_truth) # ./data/publishing_<hash>_ground_truth.parquet
print(r.total_records) # ~10150 (size + dups + hard negatives)
```
### CLI
```bash
# Minimal
dupehell --domain kyc --size 100000 --seed 42
# Full options
dupehell --domain kyc --size 1000000 --seed 42 \
--difficulty hell --output-format parquet --output-dir ./output
# Help
dupehell --help
```
### Graph output
Add `--graph` (CLI) or `generate_graph=True` (Python) to also emit a property
graph — nodes (one per record) and typed edges (`fk`, `exact_dup`,
`fuzzy_dup`, `hard_neg`) linking `record_id`s that a record-linkage/graph
pipeline should (or shouldn't) merge:
```bash
dupehell --domain fintech --size 10000 --seed 42 --graph
```
```python
r = generate(domain="fintech", size=10000, seed=42, generate_graph=True)
print(r.nodes) # ./fintech_<hash>_nodes.parquet
print(r.edges) # ./fintech_<hash>_edges.parquet
```
See the [README](../README.md#graph-generation) for more on the graph schema.
---
## Output
Each run produces:
- `{domain}_{hash}.parquet` — main dataset
- `{domain}_{hash}_ground_truth.parquet` — ground-truth labels
- `{domain}_{hash}_nodes.parquet` / `{domain}_{hash}_edges.parquet` — property
graph (only with `--graph` / `generate_graph=True`)
| Parquet | `.parquet` | Default, ZSTD compressed |
| IPC (Arrow) | `.ipc` | Via `--output-format ipc` (or `--graph-format ipc` for the graph files), fastest write |
### `{entity}_id` columns are structural keys, not attributes to match on
Every entity carries a primary-key column named `{entity}_id` (e.g.
`researcher_id`, `vehicle_id`). It plays two roles at once:
1. It is the foreign key that ties an entity's flattened child tables back
together (e.g. `publication.researcher_id` → `researcher.researcher_id`).
2. It is also emitted as a plain column on the entity's own rows.
Because of role (1), this column is **never** noised and stays byte-identical
across every duplicate of the same `master_id`, including at `hell`
difficulty — noising it would break the join to the entity's child tables.
This means `{entity}_id` is not a realistic stand-in for a cross-source
record-linkage attribute: two independent source systems would never share
the same internal ID. If you feed it into an ER model as a candidate
matching field, you will get a free, unrealistic signal that inflates
recall/precision relative to what a true cross-source scenario would allow.
**Exclude `{entity}_id` columns from ER feature/blocking inputs** — use
`record_id` (which does vary per duplicate row) as the row identifier
instead, and treat `{entity}_id` purely as a structural join key for
reassembling an entity's child tables.
---
## Schema validation
Schemas are validated with Pydantic before generation:
```python
from dupehell import load_and_validate
schema = load_and_validate("schemas/kyc.json")
print(schema.domain) # "kyc"
print(schema.entities) # list of entity definitions
print(schema.hn_types) # hard-negative type configs
```
---
## Ethics
DupeHell generates **synthetic data** for **educational and research purposes
only** — specifically for benchmarking entity resolution algorithms.
- All data is procedurally generated — no real PII is used or distributed
- You may **not** use it for fraud, impersonation, surveillance, or any
illegal activity
- See [ETHICS.md](../ETHICS.md) for the full policy and prohibited uses
---
## Next steps
- [API reference](API.md) — full Python & Rust API documentation
- [Architecture overview](CONTRIBUTING.md) — codebase structure
- [Benchmarks](BENCHMARK.md) — performance metrics