Module duplicates 

Detecting duplicate model weights by size then a cheap content fingerprint.

Structs

DuplicateGroup

A set of files found to be duplicates of one another.

Constants

DEFAULT_THRESHOLD

The default minimum file size (256 MiB) considered for duplicate detection — small files are not worth reporting.

Functions

content_fingerprint

A content fingerprint for the file at path: the lowercase-hex SHA-256 of its first and last megabyte (or the whole file, if small). None if unreadable. Two files with the same fingerprint and size are treated as the same weights.

detect

Find groups of duplicate weight files among candidates (name, path pairs). Files smaller than threshold are ignored. Candidates are bucketed by exact size, then confirmed by a fingerprint over the first and last megabyte, so only same-size, same-fingerprint files are reported. Groups are returned most-wasteful first.