Module encoder 

Text-to-Hypervector Encoding using Hyperdimensional Computing (HDC) principles.

This module provides a deterministic text encoder that converts text strings into HVec10240 hypervectors without requiring external ML dependencies or embeddings.

Algorithm

1. Tokenize: Split on whitespace, lowercase, optional unicode segmentation
2. Token → base HVec: FNV-1a hash → seeded PRNG → random HVec10240
3. Position encoding: token_hv.permute(position * stride)
4. Bundle: Majority-rule bundling of all position-encoded token vectors
5. Optional: Character n-gram overlay for typo robustness

Hash Stability

Token hashing uses FNV-1a (Fowler–Noll–Vo 1a, 64-bit), implemented inline with no external dependencies. FNV-1a is guaranteed stable across Rust versions and platforms, unlike std::collections::hash_map::DefaultHasher (SipHash), which is explicitly documented as non-stable. This ensures encoded vectors are reproducible across Rust upgrades and different builds.

Example

use csm_core_lib::encoder::{TextEncoder, TextEncoderConfig};
use csm_core_lib::HVec10240;

let encoder = TextEncoder::new();
let hv1 = encoder.encode("hello world");
let hv2 = encoder.encode("hello world");
assert!(hv1.cosine_similarity(&hv2) > 0.99); // Deterministic

Structs

TextEncoder

Deterministic text-to-hypervector encoder using HDC principles.

TextEncoderConfig

Configuration for the text encoder.