Module dedup 

Cross-split duplicate coalescing and the deterministic exclusion record.

Implemented by plan 02-03. Cross-split duplicate coalescing and the deterministic exclusion record.

Duplicate groups are connected components over the union of exact-hash and normalized-hash edges. Grouping independently by both keys would double-count — an exact duplicate is necessarily also a normalized duplicate — and could decrement a class pool twice.

Prepare-time duplicate content is excluded and recorded, never fatal (D-18, upheld by D-27); the typed error fires only when the reduced pool can no longer supply shots_per_class.

Why coalescing is not an optimization

hash.rs proves, as a property test, that an exact-hash collision implies a normalized-hash collision. So the two edge kinds are not independent: every exact duplicate appears in BOTH groupings. Emitting one group per key would remove the same training row twice from the same class pool, understating the pool. A pool understated near the boundary produces a CrossSplitDuplicateUnderflow at selection time — a failure invented by the detector rather than present in the data.

Why nothing here returns Err

[coalesced_exclusions] returns an [ExclusionRecord], not a Result, and that is a decision rather than an omission (D-18, upheld verbatim by D-27). Hard-failing at SELECTION time would make failures seed-dependent, so a subset of benchmark cells would die and a completeness gate would reject the run for a reason unrelated to the method. Hard-failing at PREPARE time would hand upstream data quality a veto over the whole dataset. The only real failure is a reduced pool that can no longer supply the requested shots, and that is raised where the shots are known: at selection.

Structs

DetectionKinds

Which detection kinds fired inside one duplicate component.

DuplicateGroup

One connected component of duplicate content spanning at least two split roles.

ExclusionRecord

The deterministic record of everything cross-split duplication removed.