Crate pith_text 

Text fingerprints: canonicalisation, word-3-shingles and a 128-word MinHash signature (design spec §4, tier-2 text lane).

Part of the pith suite: every crate in the suite builds without a single registry package.

Pipeline

1. canonicalize: NFC (via pith_unicode::nfc) → lowercase → strip trailing whitespace → exactly one '\n'.
2. Words = split_whitespace runs of the canonical text; combining marks stay attached to their base letter (accents are kept — stripping them would erase distinctions Vietnamese relies on).
3. Shingles = consecutive k-word windows, k = min(3, n) for an n-word document; hashed with FNV-1a 64 over the words joined by a single ASCII space.
4. signature = 128-word MinHash: permutation i draws seed_i from SplitMix64::new(SEED_STREAM) and each signature word is min over shingles of sm64_mix(fnv1a64(shingle) ^ seed_i), where sm64_mix is the splitmix64 output function applied as a pure finalizer (spec §4: splitmix64(FNV1a64(w) ^ seed_i)).

The crate is no_std: only alloc containers and core string operations are used, so the same code runs on embedded targets. The std feature (on by default) links std so the cdylib the language SDKs bind through carries a panic handler.

Modules

ffi

The C ABI surface of pith-text: the entry points the Python (ctypes), Node (koffi) and Go (cgo) SDKs bind through.

reference

The canonical serialization the reference.json vectors are defined over — the single source of truth shared by the vector generator (crate::reference consumers, tools/gen-reference) and the FFI surface (crate::ffi).

Constants

SIGNATURE_WORDS

Length of a signature in u64 words.

Functions

canonicalize

Canonicalises input for fingerprinting.

jaccard_estimate

Estimates the Jaccard index of the two shingle sets that produced a and b: the fraction of equal signature words, returned as an f64 in [0.0, 1.0].

signature

The 128-word MinHash signature of input.