Expand description
Deterministic content hashing: token hashes, rolling k-grams, winnowing.
Every hash here is a pure function of token content (kind tags and normalized or raw text). No process randomness, no position, no file identity enters any hash, so runs are reproducible and equal content always collides intentionally. The 64-bit FNV hashes are used only for candidate indexing. Grouping uses a domain-separated 128-bit BLAKE3 digest, so an attacker-controlled FNV collision cannot combine unrelated findings.
Functions§
- kgram_
hashes - Rolling polynomial hashes of every k-gram of
units(mod 2^64):gram(i) = units[i]·B^(k-1) + … + units[i+k-1]. - norm_
sequence_ hash - Content key of a normalized token sequence.
- norm_
token_ hash - Hash one normalized token: kind tag, atom discriminant, atom payload.
- raw_
sequence_ hash - Content key of a raw token sequence: the fold of its per-token hashes.
- raw_
token_ hash - Hash one raw token: kind tag plus raw text.
- winnow
- Winnowing: select fingerprints from k-gram hashes.