Module normalize 

Canonical normalization shared by the text soft-binding family.

Two streams are produced from the same source text:

• canonical — the surface stream used by text-fingerprint.1 and text-minhash.1: NFC, strip zero-width/format characters, lowercase, collapse whitespace, strip punctuation. Reformatting, re-encoding, case and whitespace edits, and zero-width injection leave it unchanged.
• structural — the structure-preserving stream used by text-structure.1: NFC and strip zero-width/format only, keeping sentence/paragraph boundaries and punctuation, which are the signal.

Functions

canonical

Surface canonical stream for the fingerprint and MinHash algorithms.

is_zero_width_format

Zero-width and format characters removed by every algorithm.

structural

Structure-preserving stream: NFC and strip zero-width/format only.

words

Word tokens of the surface stream: the whitespace-separated pieces of canonical. Used by MinHash shingling and structural token counting.