Module vision_pre 

Host preprocessor for vision input (lane/vision): bytes -> ViT patch rows.

Qwen2VLImageProcessorFast semantics: smart_resize to multiples of factor = patch(16) * merge(2) = 32 with the pixel-area budget, rescale 1/255, normalize mean/std 0.5 -> [-1, 1], patchify to [ghgw, 321616 = 1536] rows in row-major grid order with (c, t, ph, pw) inner order — the flatten of the conv weight [1152, 3, 2, 16, 16], so VisionTower::forward consumes rows directly. Images duplicate their frame across temporal_patch 2; videos fill the pair with consecutive sampled frames.

Resize filter: CatmullRom (Keys bicubic a=-0.5, PIL-BICUBIC family). The HF fast processor runs torch bicubic (a=-0.75) antialias — close but not bit-equal; the merger-cosine parity gate arbitrates whether the difference matters.

Structs

PreppedImage

PreppedVideo

One prepared VIDEO: temporal groups as PreppedImage units (each = one pad run of gh*gw/4 tokens) + per-group timestamps for the HF placeholder format (<t.t seconds> before each group’s pad run). Groups forward TOGETHER through VisionTower::forward_seq — one attention span per video, the HF cu_seqlens law.

VisionUnit

One pad-run unit crossing the API boundary: a standalone image, or one temporal group of a video. Units with the same video index are consecutive and forward TOGETHER through forward_seq (one attention span per video).

Constants

MAX_PIXELS

MIN_PIXELS

Area budget (pixels) from preprocessor_config: shortest_edge / longest_edge.

VID_MAX_FRAMES

Sampled frame cap (2 frames per temporal group).

VID_MIN_PIXELS

Functions

decode_data_uri

Parse a base64 data URI into raw bytes (any data:*;base64, media type).

prep_data_uri

data:image/...;base64,<payload> -> patch rows.

prep_image_bytes

Image bytes (png/jpeg/webp/gif/bmp) -> patch rows. The single frame fills both temporal slots (HF: images are tiled to temporal_patch_size).

prep_video_gif

Animated GIF -> prepared video: decode frames + delays, uniform-sample to an even count <= VID_MAX_FRAMES, resize on the total-pixel budget, patchify CONSECUTIVE frame pairs into temporal groups (frame 2g fills t=0, 2g+1 fills t=1). Timestamps come from the GIF’s own delays at the sampled indices (HF _calculate_timestamps).

smart_resize

smart_resize (HF): round each side to a multiple of 32 preserving aspect ratio, then scale into the [MIN_PIXELS, MAX_PIXELS] area budget.

video_max_pixels

Serving cap on total video patches (groupsghgw): sdpa_naive keys the whole span in shared memory, so the pixel budget stays well under the HF default. Env-tunable.