Crate symproj 

symproj

Codebook-based token-to-vector projection.

Maps token IDs to dense vectors with a Codebook, then exposes pooled phrase vectors and per-token vector sequences.

Scope

symproj does not train embedding models or load external embedding file formats. Use it when the vocabulary and embedding matrix already exist and the remaining job is lookup, pooling, normalization, or sequence output for downstream retrieval code.

Structs

Codebook

A Codebook maps token IDs to dense vectors.

EncodeStats

Counts how many token IDs contributed to a lenient encoding.

Projection

A Projection combines a Tokenizer and a Codebook.

Enums

Error

Functions

l2_normalize_in_place

L2-normalize a vector in place.

remove_component_in_place

Remove a component direction (u) from a vector (v): [ v \leftarrow v - u \frac{u \cdot v}{u \cdot u} ]

remove_component_unit_in_place

Remove a pre-normalized component direction from a vector.

sif_weight

SIF (Smooth Inverse Frequency) weight from Arora et al. (2017): [ w(p) = \frac{a}{a + p} ] where (p) is token probability and (a) is a small smoothing constant (often (10^{-3})).

Type Aliases

Result