Struct MtmdContext 

pub struct MtmdContext { /* private fields */ }

The main multimodal context.

Wraps a mtmd_context *. This context is tied to a specific mmproj model file and a loaded LlamaModel. It is safe to share across threads for tokenize calls (read-only), but encode_chunk / eval helpers mutate internal state and must not be called concurrently.

Implementations

impl MtmdContext

pub fn default_marker() -> &'static str

Returns the default media marker string used in prompts (currently "<__media__>").

pub fn init_from_file( mmproj_path: impl AsRef<Path>, text_model: &LlamaModel, params: MtmdContextParams, ) -> Result<Self>

Initialise a multimodal context from an mmproj GGUF file.

Parameters

• mmproj_path – path to the mmproj .gguf file
• text_model – the already-loaded text model
• params – context parameters (use MtmdContextParams::default())

Errors

Returns MtmdError::ContextCreateFailed if the underlying C call returns a null pointer.

pub fn void_logs()

Silence all clip/mtmd log output by installing a no-op callback.

Call this right after init_from_file to suppress the verbose clip_model_loader: tensor[N]… lines that clip.cpp emits to its own private logger (separate from llama_log_set).

pub fn void_helper_logs()

Like void_logs, but additionally silences logs emitted by the mtmd_helper_* layer (e.g. eval/decode helpers).

Internally calls mtmd_helper_log_set which also routes through mtmd_log_set, so this is a strict superset of void_logs.

pub fn supports_vision(&self) -> bool

Returns true if the model supports vision (image) input.

pub fn supports_audio(&self) -> bool

Returns true if the model supports audio input.

pub fn supports_video(&self) -> bool

Returns true if this build and model support video input.

Video support additionally requires ffmpeg/ffprobe to be available at runtime (see MtmdVideo). Wraps mtmd_helper_support_video.

pub fn marker(&self) -> &str

Returns the media marker string configured for this context.

Unlike default_marker (the library-wide default), this reflects any override passed via MtmdContextParams::media_marker. Wraps mtmd_get_marker.

pub fn audio_sample_rate(&self) -> i32

Returns the audio sample rate in Hz (e.g. 16_000 for Whisper), or -1 if audio is not supported.

pub fn decode_use_non_causal(&self, chunk: &MtmdInputChunk<'_>) -> bool

Whether llama_decode must use a non-causal attention mask when decoding image embeddings for this model.

pub fn decode_use_mrope(&self) -> bool

Whether the model uses M-RoPE for llama_decode.

pub fn tokenize( &self, text: &MtmdInputText<'_>, bitmaps: &[&MtmdBitmap], output: &mut MtmdInputChunks, ) -> Result<()>

Tokenize a text prompt that contains one or more media markers.

The number of bitmaps must equal the number of media markers in the prompt text, otherwise MtmdError::TokenizeError with code 1 is returned.

This call is thread-safe (shared &self).

Parameters

• text – text + tokenisation options
• bitmaps – slice of MtmdBitmap references, one per media marker
• output – an MtmdInputChunks that will be populated with the result

Errors

Returns MtmdError::TokenizeError if tokenization fails.

pub fn tokenize_from_parts( &self, parts: &[MtmdInputPart<'_>], add_special: bool, output: &mut MtmdInputChunks, ) -> Result<()>

Tokenize an explicit sequence of parts, without media markers.

Self::tokenize splices bitmaps in wherever the prompt contains the media marker, which means the marker string has to be embedded in the text and cannot itself be user content. This takes the interleaving directly, so:

• a marker appearing in user text is just text, not a splice point;
• parse_special is per text part, so a system prompt can enable special tokens while user content does not.

add_special applies once to the whole sequence — upstream ignores the per-part flag.

Wraps mtmd_tokenize_from_parts.

Errors

Returns MtmdError::TokenizeError — code 1 means a part carried both text and a bitmap, or neither.

pub fn gen_audio_info(&self) -> Option<MtmdGenAudioInfo>

Audio-generation capabilities of the loaded mmproj.

Returns None when this projector cannot generate audio, which is the case for every vision-only mmproj.

Wraps mtmd_gen_audio_get_info.

pub fn model_can_chat(&self, ctx: &LlamaContext<'_>) -> bool

Whether this model and projector can be used for chat.

Wraps mtmd_helper_model_can_chat.

pub fn encode_chunk(&self, chunk: &MtmdInputChunk<'_>) -> Result<()>

Encode a single input chunk (image or audio) and store the resulting embeddings inside the context.

After a successful call, the embeddings can be retrieved with MtmdContext::output_embd.

This call is NOT thread-safe.

Errors

Returns MtmdError::EncodeError if encoding fails.

pub fn output_embd(&self, n_elements: usize) -> &[f32]

Return a slice over the embeddings produced by the last encode_chunk call.

The length (in f32 elements) is:

n_embd_inp(model)  *  chunk.n_tokens()

Safety

The returned slice is valid until the next call that mutates the context (e.g. another encode_chunk).

pub fn eval_chunks( &self, lctx: *mut llama_context, chunks: &MtmdInputChunks, n_past: i32, seq_id: i32, n_batch: i32, logits_last: bool, new_n_past: &mut i32, ) -> Result<()>

High-level helper: evaluate (decode) all chunks in sequence.

• Text chunks are decoded via llama_decode.
• Image/audio chunks are first encoded with mtmd_encode_chunk and then decoded via llama_decode.

On success new_n_past is updated with the new past position.

This call is NOT thread-safe.

Parameters

• lctx – raw pointer to the llama context (from LlamaContext::as_ptr)
• chunks – the tokenized chunks to evaluate
• n_past – current KV-cache position
• seq_id – sequence ID
• n_batch – maximum batch size (must be ≥ 1)
• logits_last – if true, compute logits only for the final token
• new_n_past – updated KV-cache position after the call

Errors

Returns MtmdError::EvalError if evaluation fails.

pub fn eval_chunk_single( &self, lctx: *mut llama_context, chunk: &MtmdInputChunk<'_>, n_past: i32, seq_id: i32, n_batch: i32, logits_last: bool, new_n_past: &mut i32, ) -> Result<()>

High-level helper: evaluate a single chunk.

Works identically to eval_chunks but operates on one chunk at a time.

Errors

Returns MtmdError::EvalError if evaluation fails.

pub fn decode_image_chunk( &self, lctx: *mut llama_context, chunk: &MtmdInputChunk<'_>, encoded_embd: &[f32], n_past: i32, seq_id: i32, n_batch: i32, new_n_past: &mut i32, ) -> Result<()>

Decode an image/audio chunk whose embeddings have already been computed (e.g. via encode_chunk followed by output_embd).

Unlike eval_chunk_single, this helper handles batching plus the non-causal-attention setup required by some models (e.g. Gemma 3, Gemma 4 audio) and the M-RoPE position layout. Use it when the embeddings are already in hand and you want the helper to take care of llama_decode plumbing.

encoded_embd must contain mtmd_image_tokens_get_n_tokens(chunk) * llama_model_n_embd_inp(model) f32 elements. This call is NOT thread-safe.

Errors

Returns MtmdError::EvalError with code -1 if chunk is not an image/audio chunk, or 1 if llama_decode fails.

pub fn as_ptr(&self) -> *mut mtmd_context

Returns a raw pointer to the underlying mtmd_context.

Safety

The returned pointer is valid for the lifetime of this MtmdContext. The caller must not free it.

Trait Implementations

impl Debug for MtmdContext

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for MtmdContext

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl Send for MtmdContext

impl Sync for MtmdContext