Expand description
Vision tower for StepFun Step-3.7-Flash (arch step35), lane/step37-vision 2026-08-30.
step37 is its OWN semantic program. Every law here was derived from the pinned
artifact (HF stepfun-ai/Step-3.7-Flash-NVFP4 @ 4275532f): config.json
vision_config (model_type perception_encoder) plus the vendor reference code the
checkpoint ships (vision_encoder.py / processing_step3.py / modeling_step3p7.py)
and the shard-header tensor census. Full census + plan:
research/step37-vision-20260830/CENSUS.md. Nothing is inherited from the qwen3_5
or gemma4 towers by analogy; the census decided this is a third program:
- ViT: 47 blocks, hidden 1536, 16 heads (head_dim 96), mlp 8960, patch 14, CLIP lineage: LayerNorm WITH biases (eps 1e-5), fused in_proj qkv (+bias), out_proj (+bias), quick_gelu MLP (x*sigmoid(1.702x); NOT gelu_tanh, NOT GEGLU), and LayerScale gammas (ls_1/ls_2) on both residual branches, which neither shipped tower has. ln_pre only (use_ln_post false, use_cls_token false).
- Positions enter twice: a learned 52x52 absolute table added to the patch embeddings (other grids: bilinear interpolation, align_corners=FALSE — the qwen3_5 table interpolates align_corners=TRUE, flagged so nobody unifies them), and a per-layer 2D rope over the FULL head_dim 96: first 48 dims rotate by the COLUMN, last 48 by the ROW, theta 10000, INTERLEAVED pairing ((2i, 2i+1) share one angle — GPT-J style; qwen and gemma both pair NeoX-style (d, d+half/2)).
- Attention: sdpa scale 1/sqrt(96), non-causal, one image or one 504-crop per segment (the reference batches tiles on the batch dim; attention never crosses tiles). No video path exists for this family.
- Head: NOT a merger. [n,1536] reshapes to [1536,g,g], runs two OVERLAPPING 3x3 stride-2 pad-1 convs (1536->3072->6144, biases), row-major flatten, then vit_large_projector 6144->4096 (no bias). 52-grid -> 169 rows, 36-grid -> 81.
- Preprocessing (processing_step3.py): CLIP mean/std, every ViT input a SQUARE
bilinear resize (728 main view, 504 crop tiles); ImagePatcher tiling for large /
extreme-aspect images (window law in
determine_window_size). The vendor Compose normalizes BEFORE resizing; per-channel affine commutes with the linear resample, memra resizes first (parity arbitrated by the fixed-pixel oracle). - Token layout per image (ids from the tokenizer, hardcoded nowhere): crops FIRST (<patch_start> + 81 pads + <patch_end>, <patch_newline> per full tile row except a trailing one), then <im_start> + 169 pads + <im_end>. Embedding rows replace pad positions in order; delimiters keep their text embeddings. Image spans are CAUSAL in the LM (standard create_causal_mask in the reference — unlike gemma4’s bidirectional islands), so the existing overlay prime path is the correct one.
v1 posture matches the shipped towers: correctness-first — f32 GEMMs (Engine::linear + add_row_inplace bias), sdpa_naive, host-side rope / LayerScale / quick_gelu / im2col; parity gate before any serving path (bin/step_vision_oracle).
Structs§
- Step
Image Plan - Tiling plan for an image of (w, h): tile count, tiles per row (x_num), and the per-tile newline mask (vendor law: a newline after each full tile row, except a trailing one on the final tile). Derivable from HEADER dims alone, so the pad run and the request’s token price are known before any canvas expands.
- Step
Vision Tower - Step
Vision Unit - One preprocessed step37 image: the 728 main view plus its 504 crop tiles, each as patch rows the tower consumes directly. Carried from the HTTP layer to the GPU worker; the patch buffers drop after the tower forward.
Constants§
- SV_
DEPTH - SV_
GRID_ MAIN - SV_
GRID_ TILE - SV_
HEADS - SV_
HEAD_ DIM - SV_
HIDDEN - SV_
IMAGE_ SIZE - Main-view edge (px) and its patch grid; crop-tile edge and grid.
- SV_
INTER - SV_
MAIN_ ROWS - Trunk rows per view after the two stride-2 downsamplers (52->26->13, 36->18->9).
- SV_
MAX_ IMAGE_ SIZE - ImagePatcher long-side cap before tiling (MAX_IMAGE_SIZE in processing_step3.py).
- SV_
PATCH - SV_
PATCH_ IN - SV_
POS_ GRID - SV_
TILE_ ROWS - SV_
TILE_ SIZE
Functions§
- step_
plan_ image - PRE-DECODE admission (hermes decode-bomb law, same as the qwen/gemma planners): header dims -> decode-budget check -> tiling plan. No canvas expands here.
- step_
prep_ image - Decode + preprocess one image: bytes -> main view + crop tiles per the vendor pipeline. Admission runs FIRST (header-only); the decoder is capped to the admitted dimensions. The returned unit’s tile count MUST match the header plan — the caller refuses on drift (pad runs are already rendered from the plan).