Enum MultiSeqKv 

pub enum MultiSeqKv<'a> {
    Contiguous(&'a mut [Vec<KvCache>]),
    Paged {
        caches: &'a mut [Vec<PagedKvCache>],
        stores: &'a SharedPagedKv,
    },
}

Where a batch of independent sequences keeps its KV.

forward_multi_seq batches the projections across sequences but must attend per sequence, because each has its own length and its own history. That per-sequence step is the ONLY place the batched path touches a cache, which is why paging it is a parameter here rather than a second copy of a 300-line function – the lesson the paged decode path taught by losing five model features one at a time to exactly that kind of copy.

Variants

Contiguous(&'a mut [Vec<KvCache>])

Paged

Fields

caches: &'a mut [Vec<PagedKvCache>]

stores: &'a SharedPagedKv

Implementations

impl MultiSeqKv<'_>

pub fn len(&self) -> usize

Sequences in the batch.

pub fn is_empty(&self) -> bool