pith-text 0.1.0

Text fingerprints: NFC canonicalisation, word 3-shingles and a 128-word MinHash signature
Documentation
  • Coverage
  • 100%
    27 out of 27 items documented0 out of 11 items with examples
  • Size
  • Source code size: 306.8 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 297.1 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 2s Average build duration of successful builds.
  • all releases: 2s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • pith-hash/pith-text
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • n24q02m

The pith suite contract

pith-text is part of the pith suite (pith-hash). Every suite repository follows the same rules; CI enforces them mechanically:

  • Naming: a library is always pith-<domain> (pith-image, pith-audio, pith-zip, ...). The curator/repository of repositories is the bare pith-hash. Never invent a second naming scheme inside the suite.
  • Version pinning: cross-library dependencies pin ~0.1 (e.g. pith-image = { version = "~0.1", path = "../pith-image" }). The whole suite moves together inside 0.1.x; breaking changes require a suite-wide version bump, never a silent minor drift.
  • Zero third-party dependencies: every crate depends only on other pith-* crates plus std. scripts/check-zero-deps.py (run in CI) fails the build on any other crate, for normal, build and dev dependencies alike.
  • No unsafe: every crate root carries #![forbid(unsafe_code)].
  • Hex-exact vectors: reference.json at the repo root is the cross-language source of truth. The gen-reference binary regenerates it; CI verifies the committed copy is current (gen-reference verify), and CD ships the regenerated file with every SDK artifact. Python, Node and Go SDKs MUST test against the same bytes.

Repository layout

src/               the pith-text library (canonicalize, signature, jaccard_estimate)
tests/fixtures/    the text conformance corpus (include_str!, LF byte contract)
tests/             conformance and reference-vector tests
tools/gen-reference  the vector generator binary (bin name: gen-reference)
reference.json     hex-exact cross-SDK test vectors

Install

Rust (the core library):

cargo add pith-text

Python / Node / Go SDKs are published from the same cdylib on every release; see the release assets or the package registries for the matching version.

Quick start

use pith_text::{canonicalize, jaccard_estimate, signature};

// NFC, lowercase and one trailing newline — every spelling converges:
assert_eq!(canonicalize("Hello WORLD"), "hello world\n");
assert_eq!(canonicalize("Ti\u{00EA}\u{0301}ng Vi\u{00EA}\u{0323}t"), "ti\u{1EBF}ng vi\u{1EC7}t\n");

// The 128-word MinHash signature over word 3-shingles:
let a = signature("The quick brown fox jumps over the lazy dog");
let b = signature("the quick brown fox jumps over the lazy cat");
assert_eq!(a.len(), 128);
let estimate = jaccard_estimate(&a, &b); // near-duplicates score high

Regenerate/verify the cross-SDK vectors:

cargo run --locked --bin gen-reference -- verify   # CI runs this
cargo run --locked --bin gen-reference -- gen      # rewrite reference.json

Contributing

See CONTRIBUTING.md.

Security

See SECURITY.md.

License

MIT © pith-hash