Struct Codebook 

pub struct Codebook {
    pub version: u32,
    pub dimensionality: usize,
    pub basis_vectors: Vec<BasisVector>,
    pub semantic_markers: Vec<SparseVec>,
    pub statistics: CodebookStatistics,
    pub salt: Option<[u8; 32]>,
}

The Codebook - acts as the private key for reconstruction

Fields

version: u32

Version for compatibility

dimensionality: usize

Dimensionality of basis vectors

basis_vectors: Vec<BasisVector>

The basis vectors forming the encoding dictionary Data is projected onto these bases

semantic_markers: Vec<SparseVec>

Semantic marker vectors for outlier detection

statistics: CodebookStatistics

Statistics for adaptive encoding

salt: Option<[u8; 32]>

Cryptographic salt for key derivation (optional)

Implementations

impl Codebook

pub fn new(dimensionality: usize) -> Self

Create a new empty codebook

pub fn with_salt(dimensionality: usize, salt: [u8; 32]) -> Self

Create a codebook with cryptographic salt for key derivation

pub fn initialize_standard_basis(&mut self)

Initialize with common basis vectors for text/binary data

pub fn initialize_byte_basis(&mut self)

Initialize with byte-level basis vectors (256 basis vectors for each byte value)

This creates a complete basis that can represent any byte data. Each byte value 0-255 gets its own basis vector.

Position basis vectors (64 vectors for positions 0-63) are also added by default. Use initialize_byte_basis_with_config to control this.

pub fn initialize_byte_basis_with_config( &mut self, include_position_basis: bool, )

Initialize with byte-level basis vectors with optional position basis

Arguments

• include_position_basis - Whether to add position-aware basis vectors (64 vectors)

pub fn train( &mut self, training_data: &[&[u8]], config: &CodebookTrainingConfig, ) -> usize

Train the codebook on representative data

This learns basis vectors by analyzing patterns in the training data. The algorithm:

1. Chunk the data into blocks
2. Find frequently occurring patterns (n-grams)
3. Create basis vectors for the most common patterns
4. Optionally add byte-level basis as fallback

Arguments

• training_data - Slice of training samples
• config - Training configuration

Returns

Number of basis vectors learned

ID Allocation

Basis vector IDs are allocated in non-overlapping ranges:

• Byte basis: 0-255
• Position basis: 256-319
• Learned patterns: 1000+

pub fn train_from_files( &mut self, paths: &[&Path], config: &CodebookTrainingConfig, ) -> Result<usize>

Train codebook from files on disk

Convenience method that reads files and trains on their content.

pub fn project(&self, data: &[u8]) -> ProjectionResult

Project data onto the codebook basis Returns coefficients, residual, and detected outliers

pub fn project_with_config( &self, data: &[u8], config: &ProjectionConfig, ) -> ProjectionResult

Project data onto the codebook using custom configuration

pub fn reconstruct( &self, projection: &ProjectionResult, expected_size: usize, ) -> Vec<u8>

Reconstruct original data from projection result

Trait Implementations

impl Clone for Codebook

fn clone(&self) -> Codebook

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Codebook

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Codebook

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Codebook

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for Codebook

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.