Struct WordSymbolizer 

pub struct WordSymbolizer { /* private fields */ }

Symbolizer for text input using word-level frequency binning.

Tokenizes on whitespace, computes word frequencies, and bins words into num_symbols frequency tiers. This detects semantic redundancy that byte-level symbolizers miss (e.g., “Home About Blog Contact” has low S_T).

Example

use epsilon_engine::symbolize::WordSymbolizer;

let text = "the quick brown fox jumps over the lazy dog the fox";
let symbolizer = WordSymbolizer::new(4);
let symbols = symbolizer.symbolize(text).unwrap();
// "the" (3×) and "fox" (2×) map to high-frequency bins
// "quick", "brown", etc. (1×) map to low-frequency bins

Implementations

impl WordSymbolizer

pub fn new(num_symbols: usize) -> Self

Create a new word symbolizer with num_symbols frequency bins.

Panics

Panics if num_symbols < 2.

pub fn symbolize(&self, text: &str) -> Result<Vec<u8>, SymbolizeError>

Symbolize text into a sequence of frequency-binned symbols.

Algorithm

1. Tokenize on whitespace (split by char::is_whitespace)
2. Count word frequencies
3. Bin words into num_symbols equal-frequency tiers
4. Map each word occurrence to its tier symbol

Errors

Returns SymbolizeError::EmptyInput if text contains no words.

Trait Implementations

impl Clone for WordSymbolizer

fn clone(&self) -> WordSymbolizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WordSymbolizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.