Struct MediaEncoder 

pub struct MediaEncoder {
    pub text: EmbeddingGemma2,
    pub proc: VisionProcessor,
    pub ids: SpecialIds,
    /* private fields */
}

The text encoder with the vision and audio towers beside it (each loaded on first use).

Fields

text: EmbeddingGemma2

proc: VisionProcessor

ids: SpecialIds

Implementations

impl MediaEncoder

pub fn load( model: &Arc<CmfModel>, pool: Option<Arc<Pool>>, ) -> Result<Self, String>

pub fn has_vision(&self) -> bool

Does the file carry the vision tower?

pub fn vision(&self) -> Result<&VisionTower, String>

The vision tower, loaded on first use (~0.7 GB of f32 weights).

pub fn warm_vision(&self) -> Result<(), String>

Load the tower now (a server does it at startup).

pub fn has_audio(&self) -> bool

Does the file carry the audio tower?

pub fn audio(&self) -> Result<&AudioTower, String>

The audio tower, loaded on first use (~1.2 GB of f32 weights).

pub fn warm_audio(&self) -> Result<(), String>

Load the audio tower now.

pub fn warm_for(&self, inputs: &[MixedInput]) -> Result<(), String>

Load the towers the inputs need (their one-time load stays out of a timed forward).

pub fn prepare_image( &self, img: &RgbFrame, budget: Option<usize>, ) -> Result<Media, String>

An image at budget soft tokens (None: the processor’s default).

pub fn prepare_frames( &self, frames: &[RgbFrame], budget: Option<usize>, ) -> Result<Media, String>

Sampled video frames at budget soft tokens a frame.

pub fn input_ids(&self, input: &MixedInput) -> Result<Vec<u32>, String>

Token ids of one input, every placeholder expanded.

pub fn layout( &self, input: &MixedInput, ) -> Result<(Vec<u32>, Placement), String>

Token ids of one input with every placeholder expanded, and where its soft tokens come from, in sequence order.

pub fn embed(&self, inputs: &[MixedInput]) -> Result<Vec<Vec<f32>>, String>

Embed inputs: unit-length vectors of the model’s width, in order.