Struct VecBatchedKvView 

pub struct VecBatchedKvView { /* private fields */ }

Simple contiguous BatchedKvView backed by a Vec<KvSlot> paired with a pool of flat key/value buffers.

Each entry i refers to slot slots[i]. The key buffer for slot i has length position * kv_dim and the value buffer likewise.

Implementations

impl VecBatchedKvView

pub fn new( slots: Vec<KvSlot>, keys: Vec<Vec<f32>>, values: Vec<Vec<f32>>, ) -> Self

Construct a new VecBatchedKvView from parallel vecs.

Panics

Panics if slots.len() != keys.len() or slots.len() != values.len().

Trait Implementations

impl BatchedKvView for VecBatchedKvView

fn slot_count(&self) -> usize

Number of concurrent request slots in this batch.

fn kv_for_slot(&self, slot: usize) -> (&[f32], &[f32])

Return the flattened key and value slices for slot slot.

fn position(&self, slot: usize) -> usize

Number of KV tokens already committed for slot slot (= the sequence position the next token will be written to).