Skip to main content

Module fingerprint

Module fingerprint 

Source
Expand description

Deterministic content hashing: token hashes, rolling k-grams, winnowing.

Every hash here is a pure function of token content (kind tags and normalized or raw text). No process randomness, no position, no file identity enters any hash, so runs are reproducible and equal content always collides intentionally. The 64-bit FNV hashes are used only for candidate indexing. Grouping uses a domain-separated 128-bit BLAKE3 digest, so an attacker-controlled FNV collision cannot combine unrelated findings.

Functions§

kgram_hashes
Rolling polynomial hashes of every k-gram of units (mod 2^64): gram(i) = units[i]·B^(k-1) + … + units[i+k-1].
norm_sequence_hash
Content key of a normalized token sequence.
norm_token_hash
Hash one normalized token: kind tag, atom discriminant, atom payload.
raw_sequence_hash
Content key of a raw token sequence: the fold of its per-token hashes.
raw_token_hash
Hash one raw token: kind tag plus raw text.
winnow
Winnowing: select fingerprints from k-gram hashes.