Expand description
Gemma-2 text ENCODER forward (the Lumina-Image 2.0 prompt encoder): token ids → per-token hidden states of every layer.
Second increment of the image-generation runtime
(docs/GENERATIVE.ru.md). A deliberately standalone, f32,
full-sequence forward — no KV cache, no sampling — loaded from a
diffusers/HF text_encoder/ directory (config.json + sharded
safetensors). Gemma-2 specifics carried exactly: embedding scale
√hidden, RMSNorm with (1 + w), sandwich norms around attention AND
the GeGLU MLP, GQA with query_pre_attn_scalar scaling, attention
logit softcapping tanh(x/50)·50, RoPE θ=10000. Sliding-window
layers are exact for prompts shorter than the 4096 window (image
prompts are), enforced by an assert.
Parity: python/gemma_ref.py + tests/textenc_parity.rs on the
real Lumina text-encoder weights.
Structs§
- Gemma
Encoder - Gemma-2 encoder: exact f32 from a diffusers directory, or CMF-quantized (mmap-resident, per-token embed dequant) from a packaged file.