Struct TextDecoder 

pub struct TextDecoder { /* private fields */ }

SmolVLM’s text tower plus its KV cache.

Implementations

impl TextDecoder

pub fn load( weights: &Path, config_json: &str, device: &Device, ) -> Result<Self, String>

Load the text tower from a checkpoint.

Errors

Propagates candle’s load errors; a missing tensor names itself, which is what a wrong prefix produces.

pub fn load_vb( vb: VarBuilder<'static>, config_json: &str, device: &Device, ) -> Result<Self, String>

Build the decoder from a VarBuilder the caller already has.

The path constructor above is written in terms of this, so a browser and a server build the same decoder from the same tensors.

One builder, cloned — not two loads. The path version used to map the file twice, once renamed for candle’s tower and once raw for ours. A VarBuilder is cheap to clone (its backend is shared), and on wasm a second load would mean a second COPY of the checkpoint in a 32-bit address space that is already the binding constraint.

pub fn load_reference( weights: &Path, config_json: &str, device: &Device, ) -> Result<Self, String>

Load with candle’s tower forced — the ORACLE arm.

examples/text_ab needs both implementations live in ONE process to compare them. The env toggle cannot do that: it is read once and cached (deliberately — a toggle re-read per call is the barrier ffai-diana’s silu paid 1.92x for). Without this constructor the A/B silently compared our tower against itself and reported a max logit delta of exactly 0.000e0, which is what a broken instrument looks like when it looks like a pass.

Errors

Same as Self::load.

pub fn reset(&mut self)

Drop everything the previous generation left in the KV cache.

Self::generate calls this unconditionally at its top, so a caller cannot forget it.

pub fn forward_embeds( &mut self, embeds: &Tensor, index_pos: usize, ) -> CandleResult<Tensor>

Logits for the LAST position, given a slice of the sequence.

index_pos is where this slice starts in the whole sequence — 0 for the prefill, then the running length. Getting it wrong does not error: RoPE simply rotates by the wrong amount and the output degrades, which is the same silent class as a mis-assembled prompt.

pub fn embed(&self, ids: &Tensor) -> CandleResult<Tensor>

Embed token ids through the tower’s own table.

pub fn generate_greedy( &mut self, inputs_embeds: &Tensor, max_new_tokens: usize, stop_ids: &[u32], ) -> CandleResult<Vec<u32>>

Greedy generation from a prefilled embedding sequence.

Deterministic by construction — argmax, no sampling, no seed needed. That is the plan’s §2 Gate 2 requirement (Decoding::Greedy is the default and the only variant that needs no seed) and it is also what makes step 5’s gate a token-equality check rather than a distribution comparison.

Errors

Propagates candle errors from the forward passes.

pub fn generate( &mut self, inputs_embeds: &Tensor, max_new_tokens: usize, stop_ids: &[u32], decoding: &Decoding, repetition_penalty: Option<f32>, ) -> CandleResult<Vec<u32>>

Generation under any Decoding strategy.

Decoding::Greedy takes the argmax path and needs no seed; Decoding::Sampled builds candle’s LogitsProcessor from the caller’s seed, so two runs with the same seed produce the same text. That is what Gate 2 bought by putting the seed in the TYPE rather than in an engine’s private state.

Errors

Propagates candle errors from the forward passes.

pub fn generate_traced( &mut self, inputs_embeds: &Tensor, max_new_tokens: usize, stop_ids: &[u32], decoding: &Decoding, repetition_penalty: Option<f32>, trace: Option<&mut DecodeTrace>, ) -> CandleResult<Vec<u32>>

Self::generate, optionally filling in a per-step timing trace.

The split it records is the one that matters for understanding VLM latency: prefill is one pass over the whole prompt, decode is one pass per token. For a VLM the prompt is mostly image tokens — 1088 of them for a single split still — so prefill is a large, fixed cost that has nothing to do with how long the answer is. Reporting a single “generation” number hides that, and hiding it is how people conclude the decoder is slow when the picture is what cost them.

Errors

Propagates candle errors from the forward passes.

pub const fn config(&self) -> &Config