Skip to main content

decoder_layer_forward

Function decoder_layer_forward 

Source
pub fn decoder_layer_forward(
    x: &mut [f32],
    layer: &DecoderLayer,
    cfg: &AttnConfig,
    rope: &RopeCache,
    enc_kv: &KvCache,
    dec_kv_self: &mut KvCache,
    tmp: &mut Vec<f32>,
    normed: &mut Vec<f32>,
)
Expand description

Run a single decoder layer incrementally (one token at a time). x: current decoder token vector [d_model] — modified in-place with residual enc_kv: precomputed encoder KV (for cross-attn) dec_kv_self: decoder self-attn KV cache (appended to here) normed: scratch buffer [d_model] — pre-allocated by caller, reused for self-attn/cross-attn/ffn