Struct KvCache 

pub struct KvCache<B: Backend> {
    pub step: usize,
    /* private fields */
}

Precomputed and growing KV cache for O(n) per-step decoder inference.

cross_kv holds the encoder cross-attention K,V projections for every decoder layer. These are constant for the entire decoding of one audio chunk and are computed once by KvCache::new before the decode loop.

self_kv holds the growing decoder self-attention K,V cache; one new row is appended per forward_decoder_cached call.

Fields

step: usize

Number of tokens decoded so far (indexes into positional embedding).

Implementations

impl<B: Backend> KvCache<B>

pub fn new(model: &Whisper<B>, encoder_output: Tensor<B, 3>) -> Self

Build a cache pre-populated with cross-attention K,V from encoder_output.

The encoder K,V projections (n_layers × 2 linear ops) are computed once here and reused at every subsequent decode step instead of being recomputed each time forward_decoder is called from scratch.

Performance

Call this once per audio chunk. The cost is n_decoder_layers × 2 matrix multiplications of shape [n_encoder_frames × n_text_state].