Module mimo_vision 

MiMo-V2.6 vision: image and video preprocessing, prompt-id expansion for the image/video placeholders, and the MiMo ViT (visual.*).

Reference semantics are the upstream processor (MiMoProcessor) and the HF MiMoVisionTransformer, with the merger taken as vLLM/sglang build it (RMSNorm, no biases — the checkpoint carries only ln_q.weight and the two merger matrices). Conventions that are easy to get wrong:

• Resize is F.interpolate(bilinear, align_corners=False) on 0..255, with no antialias and no rounding back to 8 bit, then ImageNet mean/std in 0..255 units. smart_resize uses factor 32 and has an upscale branch for a side below 32 that skips the aspect check.
• Every image is two identical frames (T = 2), and each patch row is the Conv3d kernel flattened as (c, tt, py, px).
• Rows are in merge-block order (t, block row, block col, mh, mw). Window blocks of type 1 run in column order: whole merge units are listed by (t, block col, block row), the RoPE tables are permuted the same way, and the band |i−j| ≤ 64 is taken over chunk-local indices in the CURRENT order. Frames are independent chunks in every block.
• Sinks add to key 0’s logit (HF, vLLM sinks_bias_key0), they are not an extra softmax column like the text model’s sinks. Beyond query 64 key 0 is masked and the sink does nothing. CMF_MIMO_VIT_SINK (key0 | column | off) switches it for A/B only.
• GQA 32/8, head_dim 64 (qk_channels), not hidden/heads = 40.

Weights are read by source name from any CMF that carries them — the <stem>.mm.cmf companion or a single-file multimodal CMF — through MimoVit::from_model. Dense (F32/F16/BF16) matrices become exact f32 GEMM operands; quantized ones (q4tp, q8_2f) stay mapped on the engine’s kernels, CPU or GPU.

The tower is unusually sensitive to weight quantization (G4.1, measured against the exact tower on 5 fixture images): q4tp gives a mean row cosine of 0.935, GPTQ-rounded q4tp 0.976, q8_2f 0.995 (CPU, int8 activations) to 0.998 (Vulkan). Every matrix group fails at q4tp on its own except the merger. The residual reaches ~4e5 in the last block, which is also why GEMM inputs are range-guarded (Lin::mm).

Structs

MimoProcessorConfig

The processor knobs, read from config.json (processor_config plus the vision geometry). Missing fields take the upstream processor’s defaults.

MimoVisionConfig

vision_config geometry.

MimoVit

The MiMo vision transformer and its patch merger.

VisualInput

One preprocessed image or video: the patch rows fed to the ViT, the patch grid, and (video) the per-frame timestamps in seconds.

Y4mInfo

The Y4M stream layout needed to seek to a frame.

Enums

SinkMode

How the per-head sinks enter the windowed blocks.

VideoSource

A silent video given as decoded frames: a directory of images with an explicit frame rate, or a Y4M stream (ffmpeg -i in.mp4 -pix_fmt yuv420p out.y4m). mp4 decoding is out of scope for v1.

VisualKind

What a visual item is; it decides the placeholder and its expansion.

Constants

AUDIO_END_ID

AUDIO_PAD_ID

AUDIO_START_ID

IMAGE_PAD_ID

MM_CONFIG_TENSOR

Name of the U8 blob that carries the checkpoint’s full config.json.

VIDEO_END_ID

VIDEO_PAD_ID

VIDEO_START_ID

VISION_END_ID

VISION_START_ID

Functions

column_permutation

Row permutation into column order: merge units listed by (t, b, a), each unit’s m² rows kept together. perm[dst] = src.

expand_prompt_ids

Expand the rendered prompt’s media placeholders.

format_timestamp

f"{int(ts // 60):02d}:{int(ts % 60):02d}" on the float32 timestamp, with torch’s float floor-division/remainder. Minutes are not wrapped.

gpu_attention_dispatches

Full-attention frame chunks this process ran through gpu::dit_attention (so a timing on a live backend cannot be mistaken for the host path).

has_vision

Whether model carries the vision tower.

prepare_image

Preprocess one image: smart_resize (factor 32), bilinear resize on 0..255, standardize, duplicate to T frames, patchify. max_pixels overrides image_max_pixels (the --image-max-pixels knob).

prepare_video

Sample, decode and preprocess a video source.

prepare_video_frames

Preprocess already-sampled frames with their timestamps: per-frame pixel budget from the sampled count, even padding (repeat the last frame and its timestamp), one smart_resize for all frames, standardize, and pair consecutive frames as temporal patches.

read_mm_config

The checkpoint config.json stored in the file as MM_CONFIG_TENSOR.

resize_bilinear

F.interpolate(x, size=(out_h,out_w), mode="bilinear", align_corners=False) on a planar [c][h][w] f32 image, no antialias. Taps nest as torch does: interpolate along W on the two source rows, then along H.

sample_frames

The sampled frame indices and their timestamps: unique(int64(linspace(0, total−1, n))) and float32(idx) / fps_src.

smart_nframes

Frame count for total source frames at video_fps (sglang smart_nframes with the processor defaults): total / fps_src · fps, clamped to [min_frames, max_frames] and total, floored to even.

smart_resize

MiMo’s smart_resize (processor MiMoProcessor.smart_resize), returning (height, width). Note the upscale branch: when the short side is below factor both sides are scaled up first and the aspect check is skipped.

video_max_pixels

Per-frame pixel ceiling for n_sampled frames: max(min_pixels, min(total_max · T // n, max_pixels)).