Carries the running counters that record_id/master_id generation
depend on across a sequence of chunked runs (--chunk-size), so chunk
N+1 continues exactly where chunk N left off instead of restarting at
0 — genuinely contiguous, global identifiers over the whole logical
dataset, with no shard-tag hack needed on top. Defaults (all zero) give
the single-run behavior, unchanged.
Generates total_size records by internally looping over
sequential, independently-seeded chunk runs (see chunk_sizes),
keeping record_id/master_id globally
contiguous across chunks (see ChunkOffsets) instead of the shard-tag
hack an external concat script would need, then assembling the chunk
outputs into one final dataset/GT/graph file via concat_dataset_files.
RAM stays bounded to roughly one --size chunk_size run at a time — each
chunk is a fully independent pipeline run, only the identifier counters
and (for the final assembly step) the output files carry over.
The output format’s hard structural limits, checked before a run
instead of discovering them as silent corruption mid-run: fixed-width
identifiers that would wrap around (duplicate ids) and --graph’s
32-bit cluster offsets. Generation itself no longer needs RAM in
proportion to --size (ground truth is streamed per batch), so these —
plus disk space, see estimate_output_bytes — are what bounds a run.
Sizes of the chunks run_chunked splits total into: chunk_size
each, except that a trailing remainder smaller than MIN_RUN_SIZE is
folded into the last full chunk instead of becoming a chunk of its own
(which build_pipeline_config would reject, after every previous chunk
had already been generated). Requires chunk_size >= MIN_RUN_SIZE.
Estimated peak disk usage of a run, in bytes: dataset + ground truth +
graph (whose IPC files exist even in Parquet mode, until converted),
doubled for --chunk-size (chunk files stay on disk until assembled).
Calibrated on real runs (see the constants above); used for the
up-front free-space check, backed by disk::DiskGuard during the run.
Upper bound on the rows --graph keeps in the ground-truth cluster map
(every written row of a duplicated master): all base + duplicate rows.
Each costs ~24 bytes of RAM while the map is built (gt::ClusterCsr).
Same as run_pipeline_with_progress, but threads record_id/
master_id counters through offsets (None == all-zero, identical to
a normal single run) and returns the counters’ final values alongside
the usual output — the mechanism --chunk-size orchestration uses to
keep IDs globally contiguous across a sequence of otherwise-independent
chunk runs. See ChunkOffsets for why this exists.
Same as run_pipeline, but invokes progress(records_written_so_far, approx_total) after every base-record batch is written to the dataset
file — approx_total is config.size (the requested record count; the
actual total including duplicates/HN/canary rows differs slightly, close
enough for a progress display). Callers that don’t need progress
reporting should use run_pipeline instead.
SOURCE_DATE_EPOCH (the reproducible-builds convention): when set to a
Unix timestamp, dupehell.timestamp uses it instead of the wall clock,
so two identical runs produce byte-identical files — that timestamp is
the only part of the output that otherwise depends on when it was
generated. Ok(None) when unset, Err when set to something that isn’t
a non-negative integer (the CLI rejects that up front rather than
silently falling back to the clock).