# BACKLOG - dedup
tests/adversarial/break_it.rs:442 | perf+test-truth | high | test_concurrent_dedup_transformer + test_concurrent_hasher ran 15+ MINUTES, hanging the CI suite, with Law-6 tautological asserts (`unique.len() > 0`, bare `is_ok()`). | FIXED - and the SUSPECTED root-cause was WRONG: DedupTransformer::push is O(1) (just `self.buffer.push`, verified src/transform/mod.rs:151), so 16k pushes are trivial. The real blowup was in finish_batch->find_clusters: the tests generated 16k NEAR-identical strings ("Thread X item Y"), the LSH worst case where every doc collapses into one bucket, forcing ~O(n^2) all-pairs verify_similarity (~256M x permutations ops). That O(n^2) is INHERENT to pathologically self-similar input (clustering N near-dupes genuinely needs the pairs), not a general defect, so right-sizing the test does NOT hide a bug. Rewrote both to be fast AND assert real values: test_concurrent_dedup_transformer now pushes 320 copies of exactly TWO word-disjoint docs from 8 threads and asserts unique.len()==2 (dupes collapse, distinct preserved); test_concurrent_hasher computes a single-threaded reference signature then 16x500 concurrent recomputes and asserts each == reference (proves lock-free determinism/thread-safety via MinHashSignature: PartialEq). Both finish in <1s. | status=done
src/lsh/mod.rs:228 | perf | medium | find_clusters is ~O(n^2) verify_similarity on dense/self-similar buckets (surfaced while root-causing break_it.rs:442): for each doc it queries LSH candidates and verify_similarity(doc, candidate) against every higher-id candidate; when a bucket is large (many near-duplicates) this is all-pairs. For CLUSTERING (connected components) most of those verifications are redundant - once two docs are already in the same component, an additional edge between them adds nothing. | FIXED (Law-7): replaced the adjacency-list + BFS with a path-compressed union-find threaded through the candidate loop. Before computing verify_similarity(doc, candidate) the loop now checks `uf_find(doc) == uf_find(candidate)` and SKIPS the similarity computation when they are already in the same component (the edge is redundant for connected components). Every component-MERGING edge is still verified (find differs -> verify -> union on sim>=threshold), so the partition is provably identical; only redundant intra-component verifications are pruned, cutting the dense-bucket all-pairs cost toward near-linear. Added module-private uf_find (iterative, path-compressed, panic-safe on missing keys) + uf_union (union by min root for a deterministic representative). Grouping now sorts components by min member so cluster ids are assigned deterministically (the old HashMap-seeded BFS numbered them nondeterministically; the partition was already the same). Representative=min-doc and len>1-only semantics preserved. Proving test lsh/tests.rs::mixed_corpus_partitions_into_exact_connected_components (two disjoint dup groups {0,1,2},{3,4} + singleton 5 -> exactly 2 clusters, exact membership, singleton excluded) is the differential check; existing find_clusters_detects_duplicates (5->1), dense_component_forms_one_cluster_without_losing_nodes (20->1) still green. | status=done
# --- rows below verified/fixed this session (my changes are green on the core suite) ---
src/config.rs:110 | silent-fallback | low | Config::with_num_bands silently ignores updates if signature_size is not divisible by num_bands | FIXED (no more silent no-op): with_num_bands now snaps the request to the divisor of signature_size nearest it (new nearest_divisor helper, single owner) so the value always takes effect AND tiles the signature. Builder returns Self (Law 3: can't become Result without breaking API); Config::new remains the loud fallible path. Proving tests with_num_bands_snaps_indivisible_request_to_valid_divisor + nearest_divisor_snaps_to_closest_divisor. NOTE: nearest_divisor's candidate scan is CAPPED at MAX_BAND_SEARCH=65536 (same hang-avoidance as optimize_lsh_params) so a hostile with_signature_size(huge)+with_num_bands cannot make it O(n); realistic band counts are far below the cap and divisor 1 is always in range. | status=done
src/config.rs:128 | silent-fallback | low | Config::with_signature_size silently ignores updates if signature_size is not divisible by current num_bands | FIXED: with_signature_size now rounds UP to the next multiple of num_bands (div_ceil*bands) instead of silently keeping the old size. Proving test with_signature_size_rounds_up_to_multiple_of_bands (100->112 with 16 bands, 256 preserved). | status=done
src/lib.rs:143 | bug | medium | optimize_lsh_params returns invalid LSH parameters if signature_size is not divisible by any of the hardcoded band counts | FIXED: enumerate every band count that EXACTLY divides signature_size (not the {4,8,16,32,64} shortlist, all multiples of 4). Any size not divisible by 4 matched none and returned the invalid seed (size=6->(8,0), size=50->(8,6)=48!=50). Now bands*rows==signature_size always; the discrimination score rejects degenerate 1-band/1-row divisors. Proving test test_optimize_lsh_params_valid_when_indivisible_by_hardcoded_bands (13 sizes incl primes 5/7/127). NOTE: the divisor search is CAPPED at MAX_CANDIDATE_BANDS=1024 (search_limit = signature_size.min(1024)) so it stays O(1024) instead of O(signature_size) — a naive 1..=signature_size loop HUNG on the pre-existing adversarial test optimize_lsh_params(usize::MAX, 0.9) (test_depth_adversarial.rs:120). band=1 always divides and is in range, so a valid (bands,rows) is still guaranteed for any size; adversarial test now passes in 0.00s. | status=done
src/minhash.rs:63 | silent-fallback | low | MinHashSignature::band silently clamps out of bounds range instead of panicking as documented | status=done | NOT-A-BEHAVIOR-BUG (doc was the lie): three existing tests (signature_band_extraction_bounds band(7,2)->[8] & band(10,2)->empty; test_impossible_empty_band_extraction band(0,10) on empty->empty) assert the graceful CLAMP as the intended, established public contract. Changing to panic would break them + be a breaking behavior change (Law 3). The clamp is NOT a Law-10 recall hazard because LshIndex::insert validates signature.len()==num_bands*rows_per_band before slicing (lsh/mod.rs:94 fix). FIXED the real defect: corrected the false "# Panics if out of bounds" doc to describe the actual clamp + why it's safe (coherence). | status=done
src/lsh/stats.rs:39 | bug | low | LshStats casts num_bands and rows_per_band to i32 which can overflow and wrap | use powf with f64 or prevent downcasting overflow | status=done | FIXED: estimated_false_positive_rate + estimated_recall now use powf(_ as f64) instead of powi(_ as i32), removing the usize->i32 downcast that wraps to a negative (inverting) exponent above i32::MAX. Matches candidate_probability's existing powf approach (ONE-PLACE consistency). All 3 .powi sites converted. | status=done
src/lsh/mod.rs:94 | bug | medium | LshIndex::insert did not verify signature length. band_hash slices [start, start+rows) and silently CLAMPS an out-of-range end (minhash.rs:63), so a signature shorter than num_bands*rows_per_band was indexed under clamped/overlapping band hashes, corrupting recall with no error. | FIXED: validate signature.len() == num_bands*rows_per_band up front, return Error::InvalidConfig on mismatch. Proving test insert_rejects_wrong_length_signature (64-len rejected, 128-len accepted). | status=done
src/lsh/mod.rs:249 | perf | medium | BFS in find_clusters marked nodes visited on DEQUEUE, so a node shared by many edges in a dense component was pushed once PER incoming edge before being popped -> quadratic queue growth. | FIXED: mark visited on ENQUEUE via `visited.insert(neighbor)` guard (returns true only on first insertion), so each node is enqueued exactly once. Seed inserted before the loop. Correctness preserved; proving test dense_component_forms_one_cluster_without_losing_nodes (20 identical docs -> 1 cluster of 20, no node lost/duplicated). | status=done
src/shingle.rs:207 | perf | low | WordShingleIterator::next allocates a new Vec on every iteration to return the shingle | ACCEPTED: a zero-alloc slice return is impossible because `words` is owned by the iterator and changing `Item` would break the public API. The per-shingle Vec is inherent to the word-shingle contract. | status=done
src/shingle.rs:165 | modernization | low | WordShingleIterator does not implement ExactSizeIterator unlike other shingle iterators | FIXED: added `impl ExactSizeIterator for WordShingleIterator<'_> {}` right after its Iterator impl. size_hint already returns an exact (remaining, Some(remaining)) so the default len() is exact; matches ShingleIterator + HashedShingleIterator (ONE-PLACE consistency). | status=done
src/transform/mod.rs:188 | bug | high | finish_batch deduplicates all samples lacking text_field using the same empty key, causing silent data loss of subsequent samples | FIXED: match sample.get(text_field) - None (field absent) => keep each as unique (nothing to compare on); Some(text) => dedup by its exact bytes as before. Distinguishes absent field from present-but-empty. Proving tests transform::tests::samples_without_text_field_are_each_kept_not_collapsed + samples_with_present_but_empty_text_field_still_dedup_by_content. | status=done
src/transform/mod.rs:178 | silent-fallback | low | finish_batch ignores index insertion errors using let _ | FIXED (Law-10): `let _ = self.index.insert(sig); inserted = true;` swallowed a failed LSH insert AND marked the doc inserted, silently dropping it from BOTH the LSH index and the byte-exact bypass dedup. Now `inserted` is set only on `insert(..).is_ok()`; on error the doc falls through to the byte-exact bypass path (preserved + content-deduped), never lost. finish_batch returns Vec<Sample> (no error channel; converting to Result is a Law-3 API break), so fail-safe-preserve is the correct surfacing here. Lib suite 93 green. | status=done
src/cluster.rs:3 | dedup | low | ClusterInfo and DuplicateCluster structs duplicate similar grouping logic and methods | FIXED: removed `ClusterInfo`; `DuplicateCluster` is now the single cluster type. Added `is_empty()` to `DuplicateCluster` for API parity, and removed the duplicate `contains()` guard in `add()` since callers pass unique doc indices. Updated `lib.rs`, `src/lsh/tests.rs`, and `tests/unit/*` to use the unified type. | status=done
src/lsh/mod.rs:179 | dedup | performance | medium | REPOPULATE 2026-07-16: LshIndex::insert performs a linear search via contains on the bucket vector before checking bucket limit, causing needless O(N) scans on full buckets and O(N^2) total insertions for large colliding sets | remove the redundant contains check or swap the check order + a proving test asserting insertion of 100k similar documents completes in under 2 seconds | FIXED (Law-7): removed the `!entry.get().contains(&doc_id)` linear O(bucket) scan from the per-insert push. It was redundant: `doc_id` cannot already be in the bucket at that point - a fresh doc was never inserted, and a re-inserted doc_id has its old band entries removed by the `had_signature` cleanup (mod.rs ~130-143), so within one band it is pushed at most once per insert. Only the `len() < MAX_BUCKET_SIZE` cap remains, keeping the push O(1). Proving tests lsh::tests::reinserting_same_doc_id_never_duplicates_it_in_a_bucket (asserts the invariant the removed guard enforced: re-inserting doc_id 7 across 5 signatures never leaves it twice in any bucket, doc_count stays 1, latest signature resolves back to it), inserting_100k_similar_documents_completes_under_two_seconds (100k docs in near-dup groups load in ~0.9s debug, well under 2s; asserts all 100k indexed and real intra-group collisions via query). Note: the separate per-insert candidate-collection loop is inherently O(bucket) and unaffected by this fix, so the perf test keeps groups small to measure insert scalability, not candidate collection. Gate: cargo test -p dedup --lib 97 passed, EXIT=0. | status=done
src/transform/mod.rs:273 | dedup | silent-fallback | low | REPOPULATE 2026-07-16: LshIndex::new failure is silently discarded during reset, keeping the old populated index while other states are reset, leading to downstream document ID collisions | surface the error by returning a Result from reset or panicking on failure + a proving test asserting reset behavior on invalid config | FIXED (Law-10, eliminated the fallible rebuild): `reset` no longer does `if let Ok(index) = LshIndex::new(&self.config) { self.index = index; }` (which SILENTLY kept the old populated index on a build error, leaving a stale index while every other field was cleared -> ghost doc_id collisions). Added an infallible `LshIndex::clear()` that empties every bucket/signature/cluster in place while preserving the band/row structure; `reset` calls `self.index.clear()`. This is strictly better than returning Result/panicking (the acceptance's options): the config was already validated at construction, so there was never a legitimate reachable failure - removing the rebuild removes the only failure path, keeps `reset` infallible (no Law-3 API break), and cannot leave a stale index. Proving tests lsh::tests::clear_empties_the_index_leaving_no_stale_entries (populate 50 docs + cluster cache, clear, assert every bucket/signature/cluster emptied, doc_count 0, band structure preserved, and a reused index collides only with newly inserted docs - no ghosts) and STRENGTHENED transform::tests::reset_clears_state (was a no-op that asserted nothing after reset; now asserts doc_count/duplicate_count return to 0 after reset AND that re-pushing pre-reset content is treated as unique, not a ghost duplicate). Gate: cargo test -p dedup --lib 97 passed, EXIT=0. | status=donesrc/minhash.rs:184-196 | dedup | correctness | high | Claude 2026-07-17: compute_from_hashed_shingles has NO empty-input guard, unlike compute (136-150) which errors on empty data and on zero shingles. An empty `shingle_hashes` slice skips the update loop entirely and returns the untouched init vector — a signature of ALL `u32::MAX`. This is a degenerate signature with pathological behavior: `similarity()` between two all-MAX signatures = 1.0 (every value matches), so any two documents that produced empty shingle sets are reported as 100%-similar DUPLICATES (false-positive dedup, silently merging unrelated docs), and an all-MAX signature can also spuriously collide with real signatures that carry u32::MAX in some slots. The asymmetry (compute fails loud on empty, compute_from_hashed_shingles fabricates a bogus signature) is itself a bug. | Guard empty `shingle_hashes` consistently with compute: change the return to `Result<MinHashSignature>` and return `Error::EmptyDocument { index: doc_id }` (or a dedicated error) when the slice is empty. Proving test: `compute_from_hashed_shingles(&[], id)` errors, and two empty-shingle documents are NEVER reported as similarity==1.0.
src/minhash.rs:39-52 | dedup | silent-fallback | medium | Claude 2026-07-17: `similarity` returns 0.0 when `self.values.len() != other.values.len()`. Different-length signatures arise only from comparing outputs of differently-configured hashers (mismatched signature_size) — a programmer/config error — yet the function silently reports "completely dissimilar" (0.0). Two IDENTICAL documents hashed under two different signature_size configs therefore read as non-duplicates: a silent recall loss that masks the misconfiguration instead of surfacing it. (Same line also collapses the empty-signature case to 0.0, conflating "no data" with "mismatched config".) | Make the length-mismatch loud: `debug_assert_eq!` the lengths (or return `Option<f64>`/document the equal-length precondition as a hard contract), and separate the empty case from the mismatch case so a config error cannot be silently swallowed as 0.0 similarity. Proving test: comparing a 64-value and a 128-value signature is flagged (assert in debug / None), not silently 0.0.
src/minhash.rs:213-214 + :236-237 | dedup | utilization/dead-code | low | Claude 2026-07-17: `exact_jaccard_similarity` (pub fn, 214) and `expected_error` (pub fn, 237) are both marked `#[allow(dead_code)]`. A `#[allow(dead_code)]` on a PUBLIC fn is a smell that the symbol is neither used on any non-test path nor actually re-exported/reachable in the crate's public API (Review Vector 11 UTILIZATION: public symbols must be used by non-test paths or removed/made private). exact_jaccard_similarity (the ground-truth Jaccard) and expected_error (MinHash variance bound) are exactly the primitives a dedup crate should USE — e.g. to validate/telemeter MinHash estimation accuracy or to size signatures for a target error — yet they sit dead. | Decide their fate: either wire them into a real non-test path (accuracy self-check, adaptive signature sizing, a public `estimate_accuracy` API) and drop the allow, or delete them if truly unwanted. Do not leave dead public symbols behind an allow. Proving: after wiring, `rg 'exact_jaccard_similarity|expected_error'` shows a non-test caller and the `#[allow(dead_code)]` is gone.