Skip to main content

decode_kv_slice

Function decode_kv_slice 

Source
pub fn decode_kv_slice(
    encoded: &KvEncodedTensorV1,
    layer: u32,
    head: u32,
    token_start: u32,
    token_end: u32,
) -> Result<Vec<f32>>
Expand description

Decode a random-access slice of the KV tensor for a single layer, head, and token range without decoding the entire tensor.

Only pages overlapping [token_start, token_end) are visited, and within each page only blocks matching layer, head, and the token range are decoded. The decoded vectors are concatenated in ascending token order.

Currently decodes batch 0 (the common single-batch inference case).