Struct BatchEncoder 

pub struct BatchEncoder { /* private fields */ }

Batch text encoder: tokenises, embeds, pools, and normalises text.

Implementations

impl BatchEncoder

pub fn new(config: EncodingConfig) -> Self

Create a new encoder with the given configuration.

pub fn tokenize(&mut self, text: &str) -> TokenizedText

Tokenise text by splitting on whitespace, truncating to max_length, and assigning sequential IDs from a growing vocabulary.

pub fn encode_single(&mut self, text: &str) -> Vec<f32>

Encode a single text string into a 128-dimensional embedding.

Steps: tokenise → produce per-token embeddings → pool → optionally normalise.

pub fn encode_batch(&mut self, texts: &[&str]) -> EncodedBatch

Encode a slice of text strings in chunks of batch_size.

pub fn pool( token_embeddings: Vec<Vec<f32>>, strategy: &PoolingStrategy, ) -> Vec<f32>

Aggregate a list of per-token embedding vectors according to strategy.

pub fn normalize_l2(v: &mut [f32])

Normalise a vector in-place to unit L2 norm. If the norm is zero, the vector is left unchanged.

pub fn similarity(a: &[f32], b: &[f32]) -> f64

Cosine similarity between two embedding vectors. Returns 0.0 if either vector has zero norm.

pub fn vocab_size(&self) -> usize

Return the number of unique tokens in the vocabulary so far.