dupey-core 0.1.2

dupey extracts comparable text from office documents, detects exact and near-duplicate files, groups them into families, and explains which file is the best latest-version candidate.

It does not use embeddings, upload files, or delete anything.

Install

cargo install dupey

Requires Rust 1.91 or newer.

Quick start

# Scan a folder and print a readable summary
dupey scan ./documents

# Emit the stable JSON contract
dupey scan ./documents --json

# Ignore additional folder names
dupey scan ./documents --exclude-dir archive --exclude-dir scratch

# Inspect one document
dupey fingerprint ./documents/proposal.docx

# Compare two versions directly
dupey compare ./documents/proposal.docx ./documents/proposal-final.docx

scan skips common vendor, VCS, and build folders such as node_modules, .git, target, dist, and build.

What dupey detects

Relation	Meaning
`exact`	Extracted document content is identical.
`near`	Documents have high lexical overlap after format-aware extraction.
`contains`	One document substantially contains another.

Supported input:

Format	Extraction
`txt`, `md`	UTF-8 text with normalized newlines
`docx`	Paragraph text and internal modification metadata
`hwp`, `hwpx`	Comparable body text and available internal timestamps
`pptx`	Slide text, excluding speaker notes
`xlsx`	Cell values with shared-string and date handling
`pdf`	Embedded text; image-only scans are reported but not compared

How it works

document
  -> format-aware text extraction
  -> normalized comparable text
       |-> SHA-256 exact hash
       `-> character shingles + MinHash
  -> exact / near / contains family
  -> explainable latest-candidate ranking

Near-duplicate detection is lexical, not semantic. This keeps results local, fast, and understandable while avoiding unrelated documents that merely share a topic.

Latest-candidate ranking

Within a family, dupey ranks files by modification time:

the document's internal modification time, when available;
otherwise, the filesystem modification time.

Filename tokens, revision counters, containment, and document length are reported as context but are not hidden ranking weights. A result is a candidate with reasons and confidence, never a claim of absolute truth.

JSON output

{
  "files": [
    {
      "path": "documents/proposal.docx",
      "format": "docx",
      "content_hash": "...",
      "fuzzy": ["..."],
      "signals": {
        "chars": 1842,
        "modified": "2026-08-20T09:30:00Z",
        "revision": 7,
        "fs_mtime": "2026-08-20T09:31:12Z"
      }
    }
  ],
  "families": [
    {
      "id": 1,
      // "exact" | "near" | "contains", or "mixed" when members joined
      // by different relations
      "relation": "mixed",
      "files": ["documents/proposal.docx", "documents/proposal-final.docx"],
      // each member names the file it actually matched, and how
      "members": [
        {
          "path": "documents/proposal-final.docx",
          "relation": "contains",
          "joined_with": "documents/proposal.docx",
          "near_score": 0.62,
          "jaccard": 0.58,
          "containment": 0.98,
          "exact_hash": "..."
        }
      ],
      // every verified pair behind this family; for "contains",
      // a is the container and b the contained document
      "edges": [
        {
          "relation": "contains",
          "a": "documents/proposal-final.docx",
          "b": "documents/proposal.docx",
          "near_score": 0.62,
          "jaccard": 0.58,
          "containment": 0.98
        }
      ],
      "pick": {
        "ranked": ["..."],
        "reasons": ["..."],
        "confidence": 0.9
      }
    }
  ],
  "errors": []
}

threshold, contains_threshold, and contains_min_jaccard are echoed at the top level so a consumer can see which gates produced the families.

The exact machine-readable schema is defined by dupey scan DIR --json.

Why near and contains have separate gates

near compares with Jaccard, whose denominator is the union of both documents. contains compares with containment, whose denominator is only the smaller document, so the same number is a far weaker bar: a shared corporate template can fill 90% of a short document without the two being versions of each other. contains therefore has its own, stricter threshold (--contains-threshold, default 0.96) plus a Jaccard floor (--contains-min-jaccard, default 0.40) that stops a short fragment quoted by many long files from chaining them into one family.

Library

The reusable engine is published as dupey-core. Its public API exposes format extraction, exact hashing, MinHash signatures, family clustering, and ranking without depending on the CLI.

Development

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace            # unit + integration (real binary, real fixtures)
python scripts/e2e.py             # cross-platform live e2e (Linux/macOS/Windows)
./scripts/e2e.sh                  # Unix convenience wrapper
cargo build --workspace --release --locked
cargo bench -p dupey-core         # criterion: extract / near_sig / cluster
./scripts/bench.sh 10             # corpus scan benchmark (10 x 100 files)

See Contributing, Direction, and Plan for project details.

Releasing

Maintainers do not edit the version manually. Run the Prepare release workflow in GitHub Actions and enter the next version without a leading v, for example 0.1.1.

The workflow updates the workspace manifest and lockfile, runs the release checks, commits and pushes the version bump to main, creates the matching v0.1.1 tag and GitHub Release, and starts the OIDC-backed crates.io publish workflow. The tag, source commit, GitHub Release, and published packages therefore all refer to the same version.

License

MIT