Module mm 

Image, video and audio attachments: decoding (files through oxideav with the media feature, decoded oxideav frames with the oxideav feature), encoding with the vision / audio towers into soft tokens, and mapping them onto the placeholder tokens of the prompt. Gemma 4 takes images, videos (frames as images with mm:ss timestamps) and audio; Qwen3.5 / 3.6 / 3.8 take images and videos (temporal patches of two frames with <t seconds> timestamps, placed with M-RoPE).

Structs

MmFeature

Soft tokens for one attachment: placeholder token id and [n, hidden] embeddings.

MmFrames

Decoded attachments (the oxideav feature) for prepare_frames: each kind goes after the files of the same kind in MmRequest.

MmRequest

Attachments for one prompt.

Functions

bind

Prepares a prompt’s attachments for a prefill at the session’s current position: declares the vision spans (Qwen M-RoPE) on sess and returns the embedding overrides for generate::prefill.

decode_images

Decodes encoded images (PNG, JPEG, WebP, GIF, BMP; the media feature).

encode_images

encode_rgb_images for encoded images (PNG, JPEG, WebP, GIF, BMP bytes; media feature).

encode_rgb_images

Images for a Qwen3.5-family vision checkpoint in model: per image, its placeholder text (<|vision_start|><|image_pad|>...<|vision_end|>, to place anywhere in a prompt) and its feature. max_soft_tokens as in MmRequest. For prompts that are not chat turns (typed decisions).

overrides

Maps placeholder token positions in ids to soft-token embeddings, in attachment order.

prepare

Decodes and encodes all attachments. Returns the placeholder text to prepend to the user message and the features in the same order. Files need the media feature.

prepare_frames

prepare with decoded frames too, each kind after the files of args (the oxideav feature; the files still need media).

vision_spans

Qwen M-RoPE: (position in ids, rows, cols) of each run of placeholder tokens (see MmFeature::grids), to declare with QwenSession::add_vision_span before the prefill.