Struct TextEncoder 

pub struct TextEncoder { /* private fields */ }

Deterministic text-to-hypervector encoder using HDC principles.

Produces consistent HVec10240 vectors from text input without external dependencies. The encoding is:

• Deterministic: Same input always produces same output
• Similarity-preserving: Similar texts produce similar vectors
• WASM-compatible: No external dependencies

Implementations

impl TextEncoder

pub fn new() -> Self

Create a new encoder with default configuration.

pub const fn with_config(config: TextEncoderConfig) -> Self

Create an encoder with custom configuration.

pub fn new_code_aware() -> Self

Create a code-aware encoder with character trigram overlay. This is the recommended configuration for CLI memory-context integration.

pub const fn config(&self) -> &TextEncoderConfig

Get the encoder configuration.

pub fn encode(&self, text: &str) -> HVec10240

Encode text into a hypervector.

The encoding process:

1. Tokenize (whitespace split, optional lowercase, optional code-aware)
2. Generate deterministic base vector for each token
3. Apply position encoding via permutation
4. Bundle all position-encoded vectors
5. Optionally add n-gram overlay

pub fn encode_with_ngrams(&self, text: &str, n: usize) -> HVec10240

Encode text with character n-grams for typo robustness.

This is equivalent to setting ngram_size in the config.

pub fn tokenize(text: &str, code_aware: bool, lowercase: bool) -> Vec<String>

Tokenize text into a vector of tokens.

This is a convenience function for reuse by other modules that need tokenization consistent with the encoder’s logic.

Arguments

• text - Input text to tokenize
• code_aware - Enable code-aware splitting (on _, -, ., /, ::)
• lowercase - Convert tokens to lowercase

Trait Implementations

impl Clone for TextEncoder

fn clone(&self) -> TextEncoder

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TextEncoder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TextEncoder

fn default() -> TextEncoder

Returns the “default value” for a type.