Module normalize 

Scope-local token normalization for Type-2 clone matching.

Type-2 clones differ from their siblings only by consistently renamed identifiers and changed literal values. Matching them requires a normal form in which those differences disappear while everything that identifies what the code does — keywords, operators, called APIs, paths, field names — is preserved.

Normalization is scoped: identifier numbering restarts for every slice passed to normalize, so a fragment’s normal form depends only on the fragment’s own content, never on what precedes it in the enclosing function. Scope-local first-occurrence numbering also makes the identifier bijection of a Type-2 match consistent by construction: two slices normalize equal exactly when a one-to-one rename maps one onto the other.

Preservation rules

An identifier keeps its text (is not renamed) when it looks like an external name rather than a local binding:

• it starts with an uppercase letter (types, enum variants, traits),
• it is adjacent to :: (a path segment),
• it follows . or -> (a member access: method, field, or a named return type after Rust’s ->),
• it precedes ! (a macro invocation).

These are lexical heuristics; a local binding that happens to match one (say, a closure named like a method) is preserved conservatively, trading a little recall for not conflating different APIs.

What the member-access rule costs, measured

The third rule is the one that gives up the most, and it earns it. Dropping it alone — still preserving types, paths and macro names — was run against the labelled corpora: seventy groups appear that this mode did not report before. Reading them, most are the families the labels already call something other than duplication: exhaustive match tables dispatching to one method per variant, forwarding split by a compile-time flag, option parsers reading one named field per line, and operations mirrored over a start and an end. Those all have one shape and differ only in which member they name, which is exactly what this rule refuses to look past.

It does lose real clones. Functions that walk a container by different link fields — first versus last, next versus previous — are labelled clones of each other, and this mode does not report them because ->next and ->prev survive normalization as different text. Structural mode reports all of them in one group, because its features read shape and token kinds and never read identifiers at all. That is the division the two modes are for: this one is the cheap screen and pays for its speed in recall.

A caution about the figure that measurement produces. The labels rule on about a seventh of what this mode reports, and that seventh is the part Structural also flagged — so it is where the genuine clones concentrate. Judged precision therefore rises when the rule is dropped, while what actually arrives is mostly the boilerplate above. The judged share of a biased sample is not this mode’s precision.

Structs

NormToken

A normalized token: the lexical kind tag plus the normalized payload.

Resolution

What a compiler resolved the names in a file to, by the byte each name starts at.

Enums

LiteralNorm

Literal-normalization strategy.

NormAtom

The normalized payload of one token.

Functions

normalize

Normalize tokens as one scope.

normalize_into

normalize into a caller-owned buffer.

normalize_resolved_into

normalize_into with what a compiler resolved about the names.