Struct CrossModalTokenizer 

pub struct CrossModalTokenizer { /* private fields */ }

Unified cross-modal tokenizer.

Manages one ModalityTokenizer per registered modality and applies:

1. Per-modality linear projection (input → shared_dim)
2. Modality-type embedding offset (for identity disambiguation)
3. Shared alignment projection (shared_dim → shared_dim)
4. Nearest-neighbour codebook quantization

Implementations

impl CrossModalTokenizer

pub fn new(shared_dim: usize) -> TokenizerResult<Self>

Create a new cross-modal tokenizer with the given shared embedding dimension.

pub fn add_modality( &mut self, config: ModalityTokenizerConfig, ) -> TokenizerResult<()>

Register a new modality.

The config.token_dim must equal self.shared_dim.

pub fn tokenize( &self, modality: &ModalityKind, input: &Array1<f32>, ) -> TokenizerResult<CrossModalToken>

Tokenize a single-modality input.

Steps:

1. Encode raw input → shared_dim embedding (per-modality encoder + GELU)
2. Add modality-type embedding offset
3. Apply shared alignment projection
4. Quantize against per-modality codebook

pub fn tokenize_batch( &self, inputs: &[(ModalityKind, Array1<f32>)], ) -> TokenizerResult<CrossModalSequence>

Tokenize a batch of (modality, signal) pairs and return them as a CrossModalSequence.

pub fn decode(&self, token: &CrossModalToken) -> TokenizerResult<Array1<f32>>

Decode a CrossModalToken back to the raw input space.

Uses the per-modality codebook entry as the quantized embedding, inverts the shared projection, removes the modality offset, and applies the pseudo-inverse decoder.

pub fn shared_dim(&self) -> usize

The shared embedding dimension.

pub fn num_modalities(&self) -> usize

Number of registered modalities.

pub fn modality_names(&self) -> Vec<String>

Sorted list of registered modality keys.

pub fn robotics_preset() -> TokenizerResult<Self>

Robotics preset: audio (16-dim), control (6-dim), sensor (9-dim) → shared_dim 64.

pub fn audio_video_preset() -> TokenizerResult<Self>

Audio-video preset: audio (80-dim), video (512-dim) → shared_dim 256.

Trait Implementations

impl SignalTokenizer for CrossModalTokenizer

SignalTokenizer implementation for CrossModalTokenizer.

Treats the input as a concatenation of registered modality signals (in registration order). Encodes each slice, concatenates the resulting embeddings, and returns the full multi-modal embedding vector.

For decode, the embedding is split back into per-modality chunks, decoded, and concatenated.

fn encode(&self, signal: &Array1<f32>) -> TokenizerResult<Array1<f32>>

Encode a concatenated multi-modal signal.

The input must be the concatenation of all registered modalities’ raw signals (in sorted key order). Each modality’s token embedding is concatenated into a single output vector of length num_modalities * shared_dim.

fn decode(&self, tokens: &Array1<f32>) -> TokenizerResult<Array1<f32>>

Decode a concatenated embedding vector back to the raw input space.

fn embed_dim(&self) -> usize

Total output embedding dimension: num_modalities * shared_dim.

fn vocab_size(&self) -> usize

Returns 0 (continuous-style tokenizer; each modality has its own discrete codebook).