Weavatrix Clone

Find duplicate code with evidence you can inspect, reproduce, and gate in CI.

weavatrix-clone is the deterministic Type-1/2/3 clone engine used by Weavatrix. It turns token-level matches into stable pairs and families with source spans, similarity proof, safety limits, JSON/SARIF output, and an explicit review boundary for semantic Type-4 candidates.

weavatrix-clone separates clone detection from search and semantic retrieval:

• exact token blocks detect Type-1 clones at arbitrary locations;
• normalized function fragments detect identifier-renamed Type-2 clones;
• bounded near-miss verification detects Type-3 clones;
• a provider trait leaves Type-4 semantic candidate generation to a separate vector/search package.

The library is MIT licensed. Its core has no dependencies. Repository scanning is an optional default feature backed by weavatrix-scan.

Why a separate crate?

Clone detection is not literal search. Search answers where a query occurs; clone detection finds related regions, verifies their relationship, groups occurrences, and returns evidence suitable for automated decisions.

The crate therefore does not depend on ripgrep, a regex engine, a vector database, Git, or an external parser process.

Detection modes

NearMiss is the library default. The explicit modes prevent an unfair speed comparison between an exact-only scan and a wider near-miss analysis.

Supported source profiles

The built-in tokenizer recognizes Rust, Go, C, C++, Bash, SQL, JavaScript, TypeScript, Python, Java, C#, and common markup. Repository defaults cover their common extensions, including JSX/TSX, module extensions, headers, and Python stubs.

These are lexical profiles, not claims of full grammar parsing. Unknown syntax still advances safely as UTF-8 tokens, and invalid UTF-8 selected source is reported rather than silently accepted.

Library use

use weavatrix_clone::{
    CloneConfig, CloneDetector, DetectionMode, Language, SourceFragment,
    SourceSpan,
};

let left = SourceFragment::new(
    "left",
    "src/a.rs",
    Language::Rust,
    SourceSpan::whole("fn value() -> u32 { 42 }"),
    "fn value() -> u32 { 42 }",
)?;
let right = SourceFragment::new(
    "right",
    "src/b.rs",
    Language::Rust,
    SourceSpan::whole("fn result() -> u32 { 42 }"),
    "fn result() -> u32 { 42 }",
)?;
let detector = CloneDetector::new(CloneConfig {
    mode: DetectionMode::Renamed,
    min_tokens: 12,
    ..CloneConfig::default()
})?;
let report = detector.detect(&[left, right])?;
# Ok::<(), weavatrix_clone::CloneError>(())

Repository scanning is available with the default scan feature:

use weavatrix_clone::{CloneDetector, RepositoryCloneDetector};

let report = RepositoryCloneDetector::new(CloneDetector::default())
    .detect(".")?;
println!("{} clone families", report.families.len());
# Ok::<(), weavatrix_clone::CloneError>(())

Exact blocks are isolated by extension and processed concurrently by default. This matches the usual language-format contract and avoids accidental cross-language hits. Set RepositoryOptions::cross_extensions or pass --cross-extensions when exact clones between formats are intentional.

Disable all optional dependencies for the fragment-only core:

[dependencies]
weavatrix-clone = { version = "0.1.3", default-features = false }

CLI

weavatrix-clone --summary --mode exact --min-tokens 24 \
  --min-lines 3 --format rs,go,ts,tsx .

The final summary includes source files, fragments, candidate comparisons, verified pairs, families, Type-1/2/3 counts, and elapsed time. Invalid options, incomplete scans, and capacity limits produce a non-zero exit code.

Machine-readable output is written to stdout and never contains timing data:

weavatrix-clone --output-format json .
weavatrix-clone --output-format sarif .
weavatrix-clone --output-format bigcloneeval /path/to/bcb/subset

json follows the checked-in versioned clone-report-v1 schema. sarif produces SARIF 2.1.0 with one primary and one related location, stable partial fingerprints, evidence properties, and Type-1/2/3 rules. bigcloneeval produces the official eight-column clone-pair CSV with no summary text.

Evidence and determinism

Every verified pair carries:

• stable source paths and byte/line spans;
• a stable pair identifier;
• clone kind and integer-backed similarity;
• strict and renamed equality flags;
• fingerprint overlap;
• bounded edit distance and compared token count.

Inputs are canonicalized before comparison and output is sorted. Hash collisions only propose candidates: exact repository windows are compared against their original token text, while structural fragments use global interned token IDs. Reordering input fragments does not reorder or renumber results.

Capacity limits cover files, tokens, candidate pairs, and repetitive hash buckets. Same-file overlapping fragments are suppressed by default.

Competitor position

This table describes contracts, not a universal accuracy ranking. jscpd has much broader format support and mature reporters; PMD has grammar-aware language integrations; SourcererCC targets Internet-scale corpora; NiCad has deep configurable pretty-printing and normalization.

Public references:

• jscpd
• PMD CPD
• SourcererCC
• Open-NiCad
• BigCloneBench

Architecture

The crate is a modular library, not a monolithic scanner:

Dependencies point inward: output and repository adapters consume the core; the core never depends on CLI or report presentation. The checked-in .weavatrix/architecture.json enforces zero runtime cycles, files no larger than 300 physical lines, functions no larger than 100 physical lines, and no grandfathered exceptions.

Correctness gates

The checked-in oracle covers:

• exact copies;
• systematic identifier renaming;
• inserted near-miss statements;
• numeric lookalikes that must remain negative;
• partial exact blocks;
• line thresholds, overlap policy, determinism, Unicode progress, invalid configuration, and duplicate fragment identifiers.

On the small Rust oracle, NearMiss detects all three intended Type-1/2/3 relations with no negative-pair hit. jscpd 5.0.12 detects the exact relation and exact subblocks in the near-miss pair, but not the identifier-renamed relation under its default exact contract. PMD CPD 7.26.0 has the same Rust oracle result even with identifier normalization.

On the portable Java oracle, Weavatrix and PMD with identifier normalization link 3/3 intended relations with no unrelated link. NiCad 7.0.1 default links 3/3 but also emits two unrelated pairs; its stricter type2 mode links 2/3 without an unrelated pair. These are relation-level smoke results, not general accuracy scores.

That oracle is a regression suite, not a published recall score. A full external benchmark remains required before making broad precision or recall claims.

AccuracyGate evaluates explicitly labeled relations with the same coverage-based location rule used by BigCloneEval. It reports overall and per-Type-1/2/3 counts, precision, recall, and F1; CI fails rather than silently accepting a missed threshold. Unlabeled detector output is deliberately ignored because an incomplete oracle cannot establish false positives.

BigCloneBench data is not bundled: its benchmark is CC BY-NC 4.0 and the IJaDataset files retain their original licenses. After obtaining the official BigCloneEval release, generate an import file with:

./benchmarks/run_bigcloneeval.sh BCB_REDUCED clones.csv
./benchmarks/run_bigcloneeval.ps1 -DatasetRoot BCB_REDUCED -OutputPath clones.csv

The runners process each functionality subset independently, use the official recommended minimum of 50 tokens and 6 lines, and emit data accepted directly by BigCloneEval's importClones.

Benchmarks

Run the dependency-free core benchmark:

cargo bench --bench core
cargo bench --bench typescript

Benchmark rules used for competitor comparisons:

• release binaries, not npx startup;
• identical language scope, minimum 24 tokens, and minimum 3 lines;
• two warmups followed by an odd number of samples;
• median wall-clock process time;
• successful exit code required for every measured run;
• Exact is compared with exact-only tools; NearMiss is reported separately.

Current local wall-clock medians in milliseconds:

* PMD skips two JavaScript shebang files and reports recoverable lexical errors. These machine-specific results are not universal. The detailed file counts, protocol, caveats, correctness tables, and reusable PowerShell runner are in benchmarks/README.md.

Current limits

• No AST binding or type resolution.
• Type-2/3 fragmentation is function-oriented; exact blocks are arbitrary.
• No bundled semantic/vector Type-4 model.
• No incremental on-disk clone index yet.
• No HTML reporter yet.
• No published BigCloneBench recall result yet.

These are explicit boundaries rather than silent fallbacks.

Development

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-targets --all-features
cargo test --no-default-features
cargo bench --bench core
cargo bench --bench typescript

License

MIT. See LICENSE.