Module reference 

The canonical serialization the reference.json vectors are defined over — the single source of truth shared by the vector generator (crate::reference consumers, tools/gen-reference) and the FFI surface (crate::ffi).

pipeline runs the documented pipeline stages over an input and reports the canonical UTF-8 bytes plus the tokenization/shingling counts; measure_signature folds a 128-word MinHash signature for compact transport (first two words verbatim, then FNV-1a 64 and SHA-256 over all 128 little-endian words). Both were moved verbatim out of tools/gen-reference/main.rs so the generator and the SDKs cannot drift; gen-reference verify compares against the committed reference.json byte-for-byte on every run.

canonical_stream packs both into the one byte stream the pith_text_fingerprint FFI hands out:

[0..4)    word_count     u32 big-endian
[4..8)    shingle_count  u32 big-endian
[8..8+C)  canonical      the canonical UTF-8 bytes (C bytes)
[8+C..)   signature      128 u64 little-endian words (1024 bytes)

so canonical_sha256 == sha256(stream[8..8+C]) and the signature folds of reference.json are folds over the 1024-byte tail. canonical_stream("") is the empty-input sentinel: zero counts, the one-byte canonical form "\n", and a tail of 128 u64::MAX words.

Structs

Canonicalized

One canonicalisation measurement: the canonical UTF-8 bytes plus the tokenization/shingling counts the pipeline derives from them.

SigFold

The 128-word MinHash signature folded for compact transport: first two words verbatim, then FNV-1a 64 and SHA-256 over all 128 little-endian words.

Constants

SHINGLE_WORDS

Shingle width of the pipeline (spec §4: three consecutive words). The signature vectors pin it implicitly: a width change drifts them and verify fails loudly.

Functions

canonical_stream

Serializes the full fingerprint of input into the canonical byte stream the reference.json vectors and the pith_text_fingerprint FFI are defined over (layout in the module docs): big-endian counts header, the canonical UTF-8 bytes, then the 128 signature words little-endian. canonical_stream("") carries the empty-input sentinel: zero counts, no canonical bytes, and a tail of 128 u64::MAX words.

measure_signature

Folds the 128-word MinHash signature of input for compact transport (see SigFold).

pipeline

Runs the documented pipeline stages over input and counts the tokens and k-word windows.