Module mtmd 

Safe wrappers for the libmtmd multimodal support library.

libmtmd extends llama.cpp with the ability to encode image and audio inputs (bitmaps) into token embeddings that can then be fed into a standard crate::context::LlamaContext::decode call alongside normal text tokens.

Quick-start

use std::path::Path;
use llama_cpp_4::{
    llama_backend::LlamaBackend,
    model::{LlamaModel, params::LlamaModelParams, AddBos},
    context::params::LlamaContextParams,
    mtmd::{MtmdContext, MtmdContextParams, MtmdBitmap, MtmdInputChunks, MtmdInputText},
};

let backend  = LlamaBackend::init().unwrap();
let model    = LlamaModel::load_from_file(&backend, Path::new("model.gguf"),
                                           &LlamaModelParams::default()).unwrap();
let mut lctx = model.new_context(&backend, LlamaContextParams::default()).unwrap();

// Load the multimodal projector (mmproj) model.
let ctx_params = MtmdContextParams::default();
let mtmd_ctx   = MtmdContext::init_from_file(Path::new("mmproj.gguf"), &model, ctx_params)
                              .unwrap();

// Load an image from a file.
let bitmap = MtmdBitmap::from_file(&mtmd_ctx, Path::new("image.jpg")).unwrap();

// Tokenize a prompt that contains the media marker.
let marker  = MtmdContext::default_marker();
let prompt  = format!("Describe this image: {marker}");
let text    = MtmdInputText::new(&prompt, true, true);
let bitmaps = [&bitmap];

let mut chunks = MtmdInputChunks::new();
mtmd_ctx.tokenize(&text, &bitmaps, &mut chunks).unwrap();

// Evaluate / decode all chunks.
let n_batch = lctx.n_batch() as i32;
let mut n_past = 0i32;
mtmd_ctx.eval_chunks(lctx.as_ptr(), &chunks, 0, 0, n_batch, true, &mut n_past).unwrap();

Feature flag

This module is only compiled when the mtmd Cargo feature is enabled.

Structs

MtmdAudioGen

Text-to-speech through an mmproj audio-generation pipeline.

MtmdAudioRequest

What to synthesize, and how.

MtmdBatch

Encode several media chunks in one pass.

MtmdBitmap

An image or audio bitmap ready for multimodal encoding.

MtmdCaps

Which modalities an mmproj file accepts.

MtmdContext

The main multimodal context.

MtmdContextParams

Parameters used when creating an MtmdContext.

MtmdDecoderPos

Per-token position used by M-RoPE decoder attention.

MtmdGenAudioInfo

What MtmdContext::gen_audio_info reports about a speech pipeline.

MtmdImageTokens

Image/audio token metadata attached to a non-text MtmdInputChunk.

MtmdInputChunk

A single tokenized input chunk (text, image, or audio).

MtmdInputChunks

A list of tokenized input chunks produced by MtmdContext::tokenize.

MtmdInputText

Text input for MtmdContext::tokenize.

MtmdLazyBitmap

A bitmap whose contents are produced on demand, during tokenization.

MtmdVideo

An open video stream, decoded frame-by-frame via ffmpeg.

MtmdVideoInfo

Metadata describing an open MtmdVideo stream.

MtmdVideoParams

Parameters controlling how a MtmdVideo stream is opened and sampled.

OwnedMtmdInputChunk

A chunk that owns its allocation, as returned by MtmdInputChunks::load_chunk or MtmdInputChunk::to_placeholder.

Enums

MtmdAudioOutType

Container for generated audio.

MtmdError

All errors that can be returned by the mtmd module.

MtmdGenAudioType

Which audio-generation pipeline an mmproj implements.

MtmdInputChunkType

The type of an MtmdInputChunk.

MtmdInputPart

One element of a marker-free prompt, for MtmdContext::tokenize_from_parts.

MtmdLazyChunk

What a lazy-bitmap callback yields for one chunk index.

MtmdVideoItem

One item read from a MtmdVideo stream by MtmdVideo::read_next.

Functions

mmproj_caps

Read an mmproj file’s input capabilities without loading it.

Type Aliases

MtmdProgressCallback

Progress callback invoked while the CLIP/mmproj weights are loading.

Result

A convenience Result alias for this module.