Module candidate 

Structural-mode candidate generation: the inverted-index seed layer.

Verifying every pair of fragments in a corpus is quadratic and hopeless at scale, so detection never starts from pairs. It starts from an inverted index: statement-window and subtree feature hashes (crate::features) map to the fragments that produced them, and only fragments that landed in the same posting list — an exact structural match under the feature recipe — become candidate pairs. The approximate near-match layer (characteristic vector nearest-neighbour, MinHash/LSH) plugs in behind the same candidate-emitting interface later; this layer is the exact-match seed.

Candidate-explosion control is a first-class concern, not an afterthought (AGENTS.md invariant 10). Two controls act here, both before any pair leaves the stage, and both count what they drop into CandidateStats rather than letting it vanish:

• high-frequency suppression — a hash whose posting list exceeds CandidateConfig::posting_cap is boilerplate-shaped noise that would dominate the pair budget; it is dropped whole and counted;
• a global candidate upper bound — posting lists are paired rarest-first, so when CandidateConfig::pair_budget runs out the high-frequency, low-signal lists are the ones sacrificed, and the set records that it was truncated.

A list is paired whole or not at all

The ceiling stops between posting lists, never inside one. That costs coverage — the list the allowance could not hold entirely is skipped rather than half-paired — and it is worth the cost because of what grouping does with a half-paired list.

Grouping treats a pair nothing proposed as a pair that is not similar (crate::grouping), which is sound while the stage above it ran to completion and is not sound once a ceiling cut a list in two. A family whose members were compared to each other only in part looks, from there, like a family whose members mostly disagree: the complete-linkage floor ejects them and the surviving comparisons come back out one by one, as pairs no group holds both halves of. One duplication that a whole list states once is then restated as many times as the ceiling happened to leave edges.

Measured against the labelled corpora with the ceiling lowered until it bites, cutting inside a list turned one twenty-seven-member family into a hundred and fifty-five pairs, and made the report grow as the ceiling came down — seven times its untruncated size at one setting, while the findings anybody had ruled on stayed exactly the same. Stopping between lists costs a few per cent of those findings at the same ceiling and leaves the rest of the report the size it was. It also makes the ceiling monotone: a run given more allowance can no longer report more.

Output is a pure function of the input: the emitted pairs are sorted deterministically, so file order only moves the file indices inside the fragment references and never changes which pairs appear or in what order.

Structs

CandidateConfig

Tuning for candidate generation.

CandidatePair

An exact-hash candidate pair: two fragments that share one feature hash.

CandidateSet

The candidate stage’s output: exact-hash pairs plus funnel statistics.

CandidateStats

Counters describing what candidate generation saw and dropped: the head of the detection funnel, recorded for the doctor/verbose view.

FragmentRef

One occurrence of a hashed fragment (a statement window or a subtree) at a source location. file indexes the slice given to generate; unit indexes that file’s FileFeatures::units.

StatementRun

Where a statement-window fragment sits in its unit’s statement sequences.

Constants

DEFAULT_PAIR_BUDGET

Default global candidate-pair upper bound. Provisional, as above.

DEFAULT_POSTING_CAP

Default posting-list cap. Provisional; the corpus funnel measurement calibrates it against real high-frequency structure.

Functions

generate

Generate exact-hash candidate pairs across files.