Expand description
Web-oriented SimHash.
This crate implements the public pieces of the Google WWW 2007 near-duplicate detection paper: weighted-feature SimHash, 64-bit Hamming comparison, and a multi-table online lookup index. It does not attempt to reproduce Google’s private crawler feature pipeline.
Structs§
- Feature
Hash - A pre-hashed feature.
- Index
Config - Match
- SimHash64
- A 64-bit SimHash fingerprint.
- SimHash
Index - SimHash
Options - Fingerprint construction options.
- Table
Spec - WebFeature
Extractor - WebFeature
Options - Weighted
Feature - A string feature with an integer weight.
Enums§
- TieBreaker
- Tie behavior when a SimHash accumulator lands exactly on zero.
Constants§
- DEFAULT_
NEAR_ DUP_ DISTANCE - The Google paper’s empirically reasonable threshold for 64-bit web-page fingerprints.
Functions§
- fnv1a64
- Deterministic 64-bit FNV-1a hash.