pelicode

English | 日本語

pelicode extracts structural features of code at the function level (hashes based on the AST) and compares them with the code of published packages to find similar code that should be reviewed.

Just as dependabot checks dependency versions against external repositories, pelicode checks the functions in your commits and pull requests against code published on repositories such as PyPI, Maven Central and crates.io. It helps you notice code from elsewhere that may have slipped in without the developers knowing, for example through AI-generated code or pull requests from outside contributors.

[!IMPORTANT] pelicode does not decide whether code is plagiarized or violates a license. It shows "similar code candidates to review", together with their sources and licenses. Whether code was reused, and whether that raises a license issue, is for a person to check and decide.

This is a research prototype (v0.1). The usage scenarios describe who pelicode is for and what works today, and the specification describes the design and the implementation.

How it works

flowchart LR
    R[("PyPI / Maven Central / crates.io<br/>Go proxy / RubyGems")] -->|pelicode crawl| I
    P["Local projects"] -->|pelicode index| I
    I["Extract, normalize and<br/>hash functions"] --> D[("SQLite DB")]
    D -->|pelicode stats --refresh| D
    T["Code to check<br/>(directory / git diff)"] -->|pelicode scan| S["Search for<br/>similar functions"]
    D --> S
    S --> O["Similar code candidates<br/>(text / JSON / CSV)"]

1. Function extraction: source files are parsed with Tree-sitter, and functions, methods, lambdas and the like are extracted one by one.
2. Normalization: superficial differences are removed step by step.
   • N0: whitespace and comments removed
   • N1: identifiers replaced with $ID, literals with $STR, $NUM and so on
   • N2: type names also replaced with $TYPE
   • N3: only the kinds of the AST nodes
3. Hashing: pelicode computes a hash of the whole function (for exact matches) and fingerprints selected by winnowing from the k-grams of the token sequence (for partial matches). MinHash signatures and LSH buckets are built from the fingerprints.
4. Search: exact matches are looked up first (N0, then N1, then N2). LSH then narrows down the candidates, which are compared by the Jaccard similarity of their fingerprint sets.
5. Filtering common code: hashes found in many packages (getters, equals, boilerplate) are no evidence of copying, so they can be ignored or down-weighted during the search.

The database never stores source code. It stores only hashes, locations (lines and byte ranges), sources (package, version, path, download URL) and licenses.

Supported languages and repositories

Installation

With Homebrew, Cargo, Docker, or from source (Rust 1.90 or later is required for Cargo and the source):

brew install tamada/tap/pelicode
cargo install pelicode-cli
docker run --rm -v "$PWD":/workdir quay.io/tama5/pelicode --help

Archives for Linux, macOS and Windows (x64 and arm64) are attached to each release. To build from source:

cargo install --path cli

This installs the pelicode command into ~/.cargo/bin. To try it without installing, use cargo run --release -- in place of pelicode. See the installation guide for details.

Quick start

# 1. Fetch published packages and index them into the database (pelicode.db)
pelicode crawl pypi requests six
pelicode crawl maven org.apache.commons:commons-lang3
pelicode crawl go github.com/google/uuid
pelicode crawl rubygems rack

# 2. Count how many packages contain each hash
pelicode stats --refresh

# 3. Check your code
pelicode scan ./src

The result looks like this:

1 function(s) have similar code candidates to review.

hex.go:10-44 parseID (tokens=275)
  similar(N1)  0.81  go:github.com/google/uuid@v1.6.0 uuid.go:68-117 [BSD-3-Clause]
               lines 12-44 <-> 70-108,110-117 (containment 0.89, 80 shared fingerprints)

The first line is the location and name of the checked function. Each following line is a candidate: the kind of match (exact(N1) for an exact match after N1 normalization, similar(N1) for a similar one), the similarity, the source and the license. The lines line shows where the matching parts are in both pieces of code.

Commands

Every command that uses a database takes --db <FILE> (default: pelicode.db); scan takes several, such as --db pypi.db --db crates.db. Run pelicode <COMMAND> --help for all options.

crawl: fetching published packages

pelicode crawl pypi requests numpy@2.1.0         # NAME[@VERSION]
pelicode crawl pypi --top 100                    # the 100 best-known packages
pelicode crawl maven com.google.code.gson:gson:2.11.0
pelicode crawl go golang.org/x/mod@v0.20.0
pelicode crawl rubygems thor@1.3.0
pelicode crawl crates uuid serde_json@1.0.140 --contact https://example.org/about
pelicode crawl maven --list packages.txt         # one package per line; '#' starts a comment
pelicode crawl pypi --updates                    # new versions of the PyPI packages already in the database

• Without a version, the latest release is fetched.
• Versions already in the database are skipped. Use --force to fetch them again.
• Archives are read in memory and discarded; nothing is unpacked to disk.
• Requests are sent one at a time, by default at least one second apart (--delay-ms). When crawling many packages, put a contact URL or e-mail address in the User-Agent with --contact.
• Failures are recorded in the database. If any package fails, the exit status is 1.
• For PyPI, the download ranking used by --top is not an official PyPI service but hugovk/top-pypi-packages. For crates.io, it is the all-time download count of crates.io. Maven Central, the Go module proxy and RubyGems publish no ranking of their own, so --top asks ecosyste.ms for them: RubyGems by downloads, Maven Central and Go, which publish no download counts, by the number of repositories that depend on a package. The aim is well-known packages, not one ranking across repositories.
• The crates.io data access policy allows at most one API request per second and strongly suggests a way to contact you; give --contact and keep --delay-ms at 1000 or more when crawling crates.io.

index: registering a local project

pelicode index ./my-project --name my-project --version 1.0 --license MIT

The ecosystem is local by default. When several directories are given, file paths are prefixed with the directory names.

stats: counting common hashes

pelicode stats --refresh          # recount, then show
pelicode stats --top 20 -f json

The counts are not updated automatically. After crawl or index, run pelicode stats --refresh, or pass --refresh-stats to crawl or index. scan warns when the counts are out of date. After the first count, a refresh recounts only the hashes of what was indexed since, with the same result as recounting everything (--refresh --full).

scan: searching for similar code

pelicode scan ./src
pelicode scan --git main..HEAD                   # only functions changed between main and HEAD
pelicode scan --git HEAD                         # only uncommitted changes in the working tree
pelicode scan --staged                           # only staged changes (for a pre-commit hook)
pelicode scan ./src -f json -o report.json
pelicode scan ./src -f sarif -o pelicode.sarif    # SARIF, for GitHub code scanning
pelicode scan ./src --exclude local:my-project   # skip your own package
pelicode scan ./src --fail-on-match              # exit with status 1 when there are candidates (for CI)

Main options:

The default --stop-packages 100 assumes a large database. With only a few dozen packages it has no effect, so give a percentage such as --stop-packages 40%.

--languages python,rust checks only some languages, and --exclude-path 'vendor/**' leaves out files by path. Candidates found only in the dependencies of the checked code (read from Cargo.lock, go.mod, requirements*.txt, poetry.lock, uv.lock, Pipfile.lock, Gemfile.lock, pom.xml and gradle.lockfile) are left out; --no-exclude-dependencies reports them too.

Settings file

Options used every time can be written in .pelicode.yaml (or .pelicode.yml, .pelicode.json5, .pelicode.json) at the root of the repository:

# .pelicode.yaml
version: 1
database: data/pelicode.db        # relative to this file
scan:
  languages: [python, rust]
  threshold: 0.6
  stop_packages: 10%
  fail_on_match: true
  exclude:
    packages: ["local:my-project"]
    paths: ["vendor/**", "**/testdata/**"]
crawl:
  contact: https://example.org/about

pelicode looks for it in the current directory and above, up to the root of the git repository. Options on the command line win over it; --config FILE names another file and --no-config ignores it. See Usage for every key.

GitHub Action

pelicode-action is a GitHub Action that checks pull requests and shows the candidates through code scanning:

# .github/workflows/pelicode.yaml
name: pelicode
on: pull_request

permissions:
  contents: read
  security-events: write   # to upload the results to code scanning

jobs:
  pelicode:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0       # pelicode compares the pull request with its base
      - uses: tamada/pelicode-action@v0.1.0
        with:
          database-url: https://example.org/pelicode/pelicode.db

See pelicode-action for its inputs.

extract / export: looking inside

pelicode extract Foo.java --tokens N1     # also show the tokens normalized at N1
pelicode extract ./src -f csv -o functions.csv
pelicode export -f json

Feature parameters

These can be given to crawl, index and extract.

The database records the parameters it was built with, and adding to it with different parameters is an error. scan uses the parameters recorded in the database automatically.

Layout

cargo test       # tests (no network access)
cargo clippy --all-targets
just --list      # the tasks CI runs (with https://just.systems/)

See CONTRIBUTING.md for the development workflow, adding a language or a package repository, and when to bump the versions of the hashes and the database.

Not implemented yet

• Using published snapshots from the GitHub Action, and a command for evaluation experiments (pelicode eval)
• Fetching the most downloaded packages of Go, Maven Central and RubyGems

License

MIT