# API Reference
## Python API
### `generate()`
```python
def generate(
domain: str,
size: int,
seed: int = 42,
difficulty: str = "medium",
output_dir: str = ".",
locale: str = "en",
pools_dir: str | None = None,
schemas_dir: str | None = None,
output_format: str = "parquet",
hard_neg_ratio: float = 0.3,
singleton_master_fraction: float | None = None,
generate_graph: bool = False,
graph_format: str = "parquet",
only_entity: str | None = None,
chunk_size: int | None = None,
) -> GenerateResult
```
Generate a synthetic dataset for a given domain.
**Parameters:**
| `domain` | `str` | — | Domain name (e.g. `"kyc"`, `"publishing"`). Must match a file in `schemas/`. |
| `size` | `int` | — | Number of base records (before duplicates & hard negatives). At least 10; see [Limits](#limits) for the upper bounds. |
| `seed` | `int` | `42` | PRNG seed for deterministic output. |
| `difficulty` | `str` | `"medium"` | One of `"light"`, `"medium"`, `"hell"`. Controls duplicate ratios and noise intensity. |
| `output_dir` | `str` | `"."` | Directory for output `.ipc` / `.parquet` files. |
| `locale` | `str` | `"en"` | Pool locale: `"en"`, `"fr"`, `"de"`, `"es"`, `"it"`, `"pt"`. Falls back to `"en"` if unavailable in a pool file. |
| `pools_dir` | `str \| None` | `None` | Path to pool data files. If `None`, tries `./assets/pools` then the package-installed path. |
| `schemas_dir` | `str \| None` | `None` | Path to domain schema files. If `None`, tries `./schemas` then the package-installed path. |
| `output_format` | `str` | `"parquet"` | `"parquet"` (default, ZSTD compressed) or `"ipc"` (Arrow IPC). |
| `hard_neg_ratio` | `float` | `0.3` | Internal scaling knob for the hard-negative count, **not** the literal fraction of records that become hard negatives. Actual count ≈ `n_duplicates × hard_neg_ratio × 0.125`, where `n_duplicates` depends on `difficulty` (light ~0.28×`size`, medium ~0.40×`size`, hell ~0.57×`size`); at `difficulty="medium"` + default `0.3` this is ~1.5% of `size`. Use `estimate()` to see the exact count for a given value before generating. |
| `singleton_master_fraction` | `float \| None` | `None` | Fraction of masters with only one record (at most 0.70/0.60/0.80 for light/medium/hell, to leave room for the tier's doublet share). If `None` (default), uses the chosen `difficulty` tier's own value (0.50/0.30/0.10 for light/medium/hell). Passing an explicit value that differs from the tier default emits a `UserWarning`, since it makes the actual duplicate volume diverge from what `estimate_difficulty()` reports for that `difficulty`. |
| `generate_graph` | `bool` | `False` | Also emit a property graph (nodes + typed edges) alongside the tabular dataset, for graph-based entity resolution / community detection benchmarking. |
| `graph_format` | `str` | `"parquet"` | `"parquet"` or `"ipc"` — output format for the graph files. Only used when `generate_graph=True`. |
| `only_entity` | `str \| None` | `None` | If set, generate only this entity (e.g. `domain="aviation"`, `only_entity="passenger"`) instead of the whole domain. `size` applies entirely to this entity. Raises `PyValueError` if not a valid entity name for `domain`. |
| `chunk_size` | `int \| None` | `None` | **Deprecated — normally unnecessary**: a single run no longer needs RAM in proportion to `size`. Kept for compatibility (emits a `DeprecationWarning`): if set and less than `size`, generates as `ceil(size / chunk_size)` sequential chunks assembled into the same single output files, needing twice the disk space while it runs. |
**Returns:** [`GenerateResult`](#generateresult)
**Raises:** `ValueError` on an out-of-range `size` or an unsupported `difficulty` / `locale` / `output_format` / `graph_format`; `FileNotFoundError` if the domain schema doesn't exist; `ValidationError` (Pydantic) if the schema is invalid; `PyValueError` (Rust) if generation itself fails.
---
### `GenerateResult`
Returned by `generate()`.
| `dataset` | `str` | Path to the generated dataset file |
| `ground_truth` | `str` | Path to the ground-truth labels file |
| `total_records` | `int` | Total records in the dataset |
| `exact_dups` | `int` | Exact duplicate rows |
| `fuzzy_dups` | `int` | Fuzzy (noised) duplicate rows |
| `hard_negs` | `int` | Hard negative pairs |
| `uniques` | `int` | Unique / singleton records |
| `masters` | `int` | Distinct master entities |
| `nodes` | `str \| None` | Path to the graph nodes file (only when `generate_graph=True`) |
| `edges` | `str \| None` | Path to the graph edges file (only when `generate_graph=True`) |
---
### `estimate_difficulty()`
```python
def estimate_difficulty(
domain: str,
size: int = 1_000_000,
seed: int = 42,
difficulty: str = "medium",
schemas_dir: str | None = None,
hard_neg_ratio: float = 0.3,
) -> DifficultyReport
```
Estimate the theoretical maximum F1 score for a given configuration without
generating data.
**Parameters:**
| `domain` | `str` | — | Domain name |
| `size` | `int` | `1000000` | Number of base records |
| `seed` | `int` | `42` | PRNG seed |
| `difficulty` | `str` | `"medium"` | `light`, `medium`, or `hell` |
| `schemas_dir` | `str \| None` | `None` | Path to schema files. If `None`, tries `./schemas` then the package-installed path. |
| `hard_neg_ratio` | `float` | `0.3` | Same internal scaling knob as in `generate()` (actual count ≈ `n_duplicates × hard_neg_ratio × 0.125`, see above) — must match the value used for actual generation for the estimate to be meaningful. |
**Returns:** [`DifficultyReport`](#difficultyreport)
---
### `load_and_validate()`
```python
Load and validate a domain schema file with Pydantic.
**Parameters:**
| `path` | `str` or `Path` | Path to a schema JSON file |
**Returns:** [`DomainSchema`](#domainschema)
**Raises:** `FileNotFoundError`, `ValidationError`
---
### Models
#### `DomainSchema`
| `domain` | `str` | Domain name |
| `entities` | `list[EntitySchema]` | Entity definitions (min 1) |
| `hn_types` | `list[HnSchema]` | Hard-negative type configurations |
#### `EntitySchema`
| `name` | `str` | Entity name |
| `columns` | `list[ColumnDef]` | Column definitions (min 1) |
| `fk_remaps` | `list[FkRemap]` | Foreign key remapping rules |
| `weight` | `float` | Relative population size of this entity within the domain (default `1.0`, equal shares) |
#### `ColumnDef`
| `name` | `str` | — | Column name |
| `type` | `str` | `"string"` | One of `string`, `int`, `float`, `boolean`, `date`, `datetime` |
| `pool_name` | `str` or `None` | `None` | Pool for random value selection |
| `nullable` | `bool` | `True` | Whether the column allows nulls |
| `null_rate_default` | `float` | `0.0` | Default null rate |
| `conditions` | `list[ColCondition]` | `[]` | Conditional column logic |
#### `ColCondition`
| `depends_on` | `str` | Name of the dependency column |
| `op` | `str` | One of `eq`, `ne`, `in`, `not_in`, `gt`, `gte`, `lt`, `lte` |
| `value` | `Any` | Comparison value |
| `action` | `str` | One of `set_null`, `set_value`, `set_pool` |
| `action_value` | `Any` or `None` | Value to apply (for `set_value` / `set_pool`) |
#### `FkRemap`
| `source_col` | `str` | Source column to remap |
| `target_entity` | `str` | Target entity to reference |
#### `HnSchema`
| `name` | `str` | Name of this HN type (informational) |
| `entity_type` | `str` | Entity type for hard negatives |
| `config_json` | `str` | JSON string with HN configuration |
---
### `DifficultyReport`
| `domain` | `str` | Domain name |
| `difficulty` | `str` | Difficulty level |
| `size` | `int` | Requested record count |
| `total_true_pairs` | `int` | Number of true duplicate pairs |
| `total_hard_neg_pairs` | `int` | Number of hard-negative pairs |
| `total_guaranteed_fp` | `int` | Estimated unavoidable false positives |
| `total_guaranteed_fn` | `int` | Estimated unavoidable false negatives |
| `precision_max` | `float` | Theoretical maximum precision |
| `recall_max` | `float` | Theoretical maximum recall |
| `f1_max` | `float` | Theoretical maximum F1 score |
| `entities` | `list[EntityDifficulty]` | Per-entity breakdown |
#### `EntityDifficulty`
| `name` | `str` | Entity name |
| `n_base` | `int` | Unique entity count |
| `n_dup` | `int` | Duplicate record count |
| `true_pairs` | `int` | True duplicate pairs |
| `hard_neg_pairs` | `int` | Hard-negative pairs targeting this entity |
| `guaranteed_fp` | `int` | Estimated unavoidable FP |
| `guaranteed_fn` | `int` | Estimated unavoidable FN |
| `columns` | `list[ColReliability]` | Per-column reliability scores |
#### `ColReliability`
| `name` | `str` | Column name |
| `col_type` | `str` | Column data type |
| `noise_damage` | `float` | Probability this column is corrupted by noise (0–1) |
| `hn_risk` | `float` | Whether this column is a hard-negative ID field (0 or 1) |
| `reliability` | `float` | Combined reliability for matching (0–1) |
---
## How Difficulty Estimation Works
`estimate_difficulty()` computes **theoretical maximum F1** without generating data:
- **`guaranteed_fp`**: hard-negative pairs that are identical on all non-HN columns — no algorithm can distinguish them from true duplicates without external information
- **`guaranteed_fn`**: true duplicate pairs where noise has destroyed discriminating columns — even perfect matching will miss them
- **`precision_max`**: derived from `total_true_pairs / (total_true_pairs + guaranteed_fp)` — ceiling on precision
- **`recall_max`**: derived from `(total_true_pairs - guaranteed_fn) / total_true_pairs` — ceiling on recall
- **`f1_max`**: harmonic mean of `precision_max` and `recall_max`
This lets you check whether your ER pipeline can "leave points on the table" or is hitting the theoretical ceiling.
**Example:**
```python
from dupehell import estimate_difficulty
report = estimate_difficulty(domain="publishing", size=1_000_000, difficulty="hell")
print(f"Max F1: {report.f1_max:.3f}") # 0.909
print(f"Guaranteed FP: {report.total_guaranteed_fp}") # 5112
print(f"Guaranteed FN: {report.total_guaranteed_fn}") # 51745
# Per-entity breakdown (EntityDifficulty has no f1_max — only totals do)
for ed in report.entities:
print(f"{ed.name}: true_pairs={ed.true_pairs}, "
f"guaranteed_fp={ed.guaranteed_fp}, "
f"guaranteed_fn={ed.guaranteed_fn}")
```
---
## Rust API (binary)
### CLI
```bash
dupehell [OPTIONS]
```
| `--domain <DOMAIN>` | `kyc` | Domain schema to use |
| `--size <SIZE>` | `1000000` | Number of base records (at least 10; see [Limits](#limits)) |
| `--seed <SEED>` | `42` | PRNG seed |
| `--difficulty <LEVEL>` | `medium` | `light`, `medium`, or `hell` |
| `--estimate` | — | Estimate theoretical max F1 and exit (no data) |
| `--pcore-only` | off | Pin the process to performance (P) cores only, on hybrid P-core/E-core CPUs (Windows only; no-op elsewhere) |
| `--output-format <FMT>` | `parquet` | `parquet` (ZSTD compressed) or `ipc` |
| `--parquet` | — | Shorthand for `--output-format parquet` (the default; kept for backward compatibility) |
| `--graph` | off | Also emit a property graph (nodes + typed edges) alongside the dataset |
| `--graph-format <FMT>` | `parquet` | `parquet` or `ipc`, only used with `--graph` |
| `--output-dir <PATH>` | `.` | Output directory |
| `--hard-neg-ratio <FLOAT>` | `0.3` | Hard negative ratio |
| `--singleton-master-fraction <FLOAT>` | tier default | Fraction of masters with only one record. Defaults to the chosen `--difficulty` tier's own value (0.50/0.30/0.10 for light/medium/hell) — pass explicitly only to override. |
| `--locale <LOCALE>` | `en` | Pool locale: `en`, `fr`, `de`, `es`, `it`, `pt` |
| `--pools-dir <PATH>` | `assets/pools` | Pool data directory |
| `--schemas-dir <PATH>` | `schemas` | Schema directory |
| `--only-entity <NAME>` | — | Generate only this entity (e.g. `--domain aviation --only-entity passenger`); `--size` applies entirely to it |
| `--chunk-size <N>` | — | **Deprecated — normally unnecessary** (a single run no longer needs RAM in proportion to `--size`). Kept for compatibility: generates in sequential chunks assembled into the same single output files, needing twice the disk space while it runs |
| `--skip-ground-truth` | off | Skip all ground-truth computation and don't write the `_ground_truth` file — for stress-test runs that only need the dataset. Incompatible with `--graph` (CLI only for now) |
### Limits
There is no fixed cap on `--size`: generation streams in fixed-size batches,
so peak RAM barely depends on it (measured 0.49 GB at 2M rows, 0.57 GB at
20M). A run is refused up front only when it would exceed:
- **Disk space** — the estimated output (dataset + ground truth + graph)
must fit in the free space of the output directory's disk; Parquet is
6-12x smaller than IPC. The run is also stopped cleanly if free space
drops below 2 GB while it runs.
- **Identifier widths** — at most 10 billion rows for one entity (its
`{entity}_id` values are 10 digits) and 1 billion hard negatives.
- **`--graph`** — at most ~4.29 billion duplicate-cluster rows. `--graph`
is also the one mode whose RAM grows with `--size` (~24 bytes per
duplicate-cluster row); a warning is printed when it may not fit.
### Reproducibility
The generated **data** depends only on the dupehell version and the
generation parameters (`domain`, `size`, `seed`, `difficulty`,
`hard_neg_ratio`, `singleton_master_fraction`, `locale`, `only_entity`,
`chunk_size`) — not on the machine, OS, or CPU core count. The only
time-dependent part of an output file is the `dupehell.timestamp`
metadata field. To make files **byte-identical** across runs and machines,
pin it with the standard `SOURCE_DATE_EPOCH` environment variable (a Unix
timestamp); this applies to both the CLI and the Python API:
```bash
SOURCE_DATE_EPOCH=1767225600 dupehell --domain aviation --size 1000000 --seed 42
```
CI checks this on every push: the same runs on Linux, Windows and macOS
must produce identical SHA-256 hashes.
### Library
The Rust crate exposes:
| `load_schema()` | `schema` | Load a domain schema from JSON |
| `build_pipeline_config()` | `schema` | Build pipeline configuration |
| `run_pipeline()` | `pipeline` | Run the generation pipeline |
| `estimate_difficulty()` | `difficulty` | Estimate theoretical max F1 |
| `Context` | `context` | Runtime context (pools, watermark) |