Struct Codebook 

pub struct Codebook { /* private fields */ }

A Codebook maps token IDs to dense vectors.

Implementations

impl Codebook

pub fn new(matrix: Vec<f32>, dim: usize) -> Result<Self>

Create a new Codebook from a flattened matrix and dimension.

pub fn get(&self, id: u32) -> Option<&[f32]>

Get the vector for a token ID.

pub fn dim(&self) -> usize

Get the embedding dimension.

pub fn vocab_size(&self) -> usize

Number of token IDs representable by this codebook.

impl Codebook

pub fn encode_ids(&self, ids: &[u32]) -> Vec<f32>

Encode a token-id sequence into a single vector using mean pooling.

This is lenient: token IDs not present in the codebook are skipped.

pub fn encode_ids_strict(&self, ids: &[u32]) -> Result<Vec<f32>>

Encode token IDs into a single vector using mean pooling (strict).

Unlike Self::encode_ids, this returns an error if any token ID is not present in the codebook. This is useful when you need a “closed vocabulary” contract.

pub fn encode_ids_weighted_strict( &self, ids: &[u32], weights: &[f32], ) -> Result<Vec<f32>>

Encode token IDs into a single vector using a weighted mean (strict).

We compute: [ v = \frac{\sum_i w_i , E[t_i]}{\sum_i w_i} ] with the convention that if (\sum_i w_i \le 0), we return the zero vector.

Weighting is one route toward SIF-style baselines (Arora et al., 2017).

pub fn encode_sequence_ids(&self, ids: &[u32]) -> Vec<Vec<f32>>

Encode token ids into a sequence of vectors (no pooling).

Trait Implementations

impl Clone for Codebook

fn clone(&self) -> Codebook

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Codebook

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.