Skip to main content

Module grouping

Module grouping 

Source
Expand description

Structural-mode clone grouping: turning verified pairs into cohesive groups.

Type-3 similarity is not transitive: A resembles B and B resembles C does not make A resemble C. Feeding verified pairs straight into a union-find and emitting the connected components would therefore fuse a chain of drifting near-clones into one incoherent group (AGENTS.md §2-9). Union-find is used here for one thing only — carving the pair graph into independent components so the expensive per-group work is bounded — and its components are never output as groups. Every component is then refined:

  1. a medoid (canonical instance) is chosen as the member with the greatest total similarity to the rest, ties broken by the smallest stable key so the choice is deterministic;
  2. the medoid constraint ejects any member too far from the medoid; the ejected members are regrouped among themselves rather than dropped;
  3. complete-linkage refinement then removes members until the weakest pair inside the group clears the cohesion floor, so every pair in a reported group — not merely every member-to-medoid edge — is similar.

Refinement is quadratic in component size, so a component past GroupingConfig::max_component is cut into pieces first and each piece refined on its own. That costs recall and never soundness — the rules that make a group cohesive are unchanged — and the count of components it fired on is reported rather than left to be inferred from the timing.

A pair that verification never proposed has no edge here; its similarity is taken as zero, which is what makes the complete-linkage floor split a chain whose ends were never compared. Singletons are not clone groups. The whole module is a pure, deterministic function of its inputs: components, candidate medoids under sampling, and every output collection are ordered by stable key, never by discovery order.

§What this asks of the stages above

Reading an absent edge as a similarity of zero is the same as saying the two were weighed and found apart. That holds while the stages above are complete per set: a family they decline to propose at all is a family nothing here claims anything about, and a family they propose is one every pair of which they proposed. It stops holding the moment a ceiling leaves a family half proposed — then a set of copies arrives looking like a set that disagrees, refinement breaks it up, and the comparisons that did survive are carried out one at a time as pairs no group holds both halves of. One duplication comes back as many, and the report grows as the allowance shrinks.

So a ceiling upstream of here has to cut between sets and never inside one. Two have been found doing otherwise — the candidate-pair budget, which used to stop in the middle of a posting list, and this module’s own GroupingConfig::max_component, which cuts a component it cannot refine whole. The first was changed to stop between posting lists; the second cannot be, since cutting is the whole point of it, so it reports which members it put apart (GroupingSet::severed_by_the_ceiling) and the caller counts those relations rather than stating them.

A ceiling that drops a whole set is fine and needs none of this: the high-frequency posting cap drops entire lists, and lowering it onto the labelled corpora only ever costs findings, never multiplies them. The distinction is not how much a ceiling removes but whether what it leaves is a set that was compared with itself.

Structs§

GroupingConfig
Tuning for grouping. Similarities are in [0, 1]; the defaults are provisional and calibrated against the chain corpus.
GroupingSet
The grouping result: refined groups plus statistics.
GroupingStats
Counters describing what grouping saw and did.
GroupingUnit
A unit as seen by grouping: only its stable key matters here.
SimilarityEdge
One verified similarity relation between two units, as produced by crate::verify. Endpoints are indices into the unit slice passed to group; the pair is undirected and a != b is required.
StructuralGroup
A cohesive clone group: a medoid plus the members that cleared both the medoid constraint and the complete-linkage floor.

Constants§

GROUPING_VERSION
Version of the rules that decide which occurrences sit in one group.

Functions§

group
Group verified pairs into cohesive clone groups.