Module vision_pre 

Host preprocessor for vision input (lane/vision): bytes -> ViT patch rows.

Qwen2VLImageProcessorFast semantics: smart_resize to multiples of factor = patch(16) * merge(2) = 32 with the pixel-area budget, rescale 1/255, normalize mean/std 0.5 -> [-1, 1], patchify to [ghgw, 321616 = 1536] rows in row-major grid order with (c, t, ph, pw) inner order — the flatten of the conv weight [1152, 3, 2, 16, 16], so VisionTower::forward consumes rows directly. Images duplicate their frame across temporal_patch 2; videos fill the pair with consecutive sampled frames.

Resize filter: CatmullRom (Keys bicubic a=-0.5, PIL-BICUBIC family). The HF fast processor runs torch bicubic (a=-0.75) antialias — close but not bit-equal; the merger-cosine parity gate arbitrates whether the difference matters.

Structs

PreppedImage

PreppedVideo

One prepared VIDEO: temporal groups as PreppedImage units (each = one pad run of gh*gw/4 tokens) + per-group timestamps for the HF placeholder format (<t.t seconds> before each group’s pad run). Groups forward TOGETHER through VisionTower::forward_seq — one attention span per video, the HF cu_seqlens law.

VisionUnit

One pad-run unit crossing the API boundary: a standalone image, or one temporal group of a video. Units with the same video index are consecutive and forward TOGETHER through forward_seq (one attention span per video).

Constants

GIF_MAX_FRAMES

Decode ceilings for prep_video_gif (hermes finding, fixed 2026-08-19): the loop used to expand EVERY frame to full-canvas RGB in host RAM before the VID_MAX_FRAMES sample — and it runs in the HTTP handler pre-admission, so a small crafted GIF (big canvas x many frames; LZW expands ~1000x) allocated GBs per request. The canvas dimensions come from the GIF header, so frames x canvas pixels is checked against the pixel ceiling AS DECODE PROCEEDS and the request is refused (clean 4xx at the handler) the moment the budget would cross — retained RAM is bounded by GIF_MAX_TOTAL_PIXELS RGB (192 MiB) plus at most one transient canvas, no matter what the stream claims. 512 frames / 67.1M px comfortably cover legitimate clips (a 480p GIF may run ~370 frames, ~12 s at 30 fps) — the serve path samples down to 32 frames and ~2M px right after this anyway.

GIF_MAX_TOTAL_PIXELS

MAX_PIXELS

MIN_PIXELS

Area budget (pixels) from preprocessor_config: shortest_edge / longest_edge.

VID_MAX_FRAMES

Sampled frame cap (2 frames per temporal group).

VID_MIN_PIXELS

Functions

decode_data_uri

Parse a base64 data URI into raw bytes (any data:*;base64, media type).

prep_data_uri

data:image/...;base64,<payload> -> patch rows.

prep_image_bytes

Image bytes (png/jpeg/webp/gif/bmp) -> patch rows. The single frame fills both temporal slots (HF: images are tiled to temporal_patch_size).

prep_video_gif

Animated GIF -> prepared video: decode frames + delays, uniform-sample to an even count <= VID_MAX_FRAMES, resize on the total-pixel budget, patchify CONSECUTIVE frame pairs into temporal groups (frame 2g fills t=0, 2g+1 fills t=1). Timestamps come from the GIF’s own delays at the sampled indices (HF _calculate_timestamps).

smart_resize

smart_resize (HF): round each side to a multiple of 32 preserving aspect ratio, then scale into the [MIN_PIXELS, MAX_PIXELS] area budget.

video_max_pixels

Serving cap on total video patches (groupsghgw): sdpa_naive keys the whole span in shared memory, so the pixel budget stays well under the HF default. Env-tunable.