Expand description
Vision tower for Qwen3.8-27B multimodal input (lane/vision, 2026-08-15).
The qwen3_5_vision ViT (depth 27, hidden 1152, heads 16, gelu_pytorch_tanh, patch 16,
spatial_merge 2, temporal_patch 2, LEARNED pos embeddings on a 48x48 grid) lives in the
official checkpoint’s outside.safetensors (the unquantized shard) — the quantized
trunks (ct-NVFP4 etc.) strip it. MEMRA_VISION_DIR points at any directory carrying
that shard; the tower output is plain [n_tokens, 5120] embeddings, so vision requests
serve on ANY trunk. Text side uses standard sequential rope (rope_scaling is null on
this model — no M-RoPE), so spliced image tokens take ordinary positions.
v1 posture: correctness-first — cuBLASLt f32 GEMMs (Engine::linear + bias epilogue),
sdpa_naive(causal=false) for the bidirectional attention, host-side permutes between
stages (the tower is a small fraction of a vision request; optimize later). Parity gate:
merger-output cosine vs the HF reference per VISION-LANE.md.
Structs§
- Embed
Overlay - Mixed-embedding prime overlay: image embeddings that replace
<|image_pad|>token embeddings at prompt-relative positions duringprime_cache_overlaid.rowsholds all images’ merger outputs concatenated ([total_rows, n_embd]); each span is(prompt_pos, row_off, n_rows)— rows[row_off, row_off+n_rows)land at prompt positions[prompt_pos, prompt_pos+n_rows). Spans must not overlap. - Vision
Tower