pub struct KvCache { /* private fields */ }Expand description
Growable key/value history for one attention layer.
Keys and values are stored post-RoPE and post-projection, laid out
[position, kv_heads * head_dim], so a decode step appends one row and the
attention math reads the whole buffer unchanged.
Implementations§
Source§impl KvCache
impl KvCache
pub fn new(kv_heads: usize, head_dim: usize) -> Self
Sourcepub fn with_capacity(kv_heads: usize, head_dim: usize, positions: usize) -> Self
pub fn with_capacity(kv_heads: usize, head_dim: usize, positions: usize) -> Self
Reserves room for positions tokens, so a generation loop does not
reallocate on every step.
Sourcepub fn len(&self) -> usize
pub fn len(&self) -> usize
Number of cached positions, which is also the absolute position of the next token to be appended.
pub fn is_empty(&self) -> bool
Sourcepub fn truncate(&mut self, positions: usize)
pub fn truncate(&mut self, positions: usize)
Keeps the first positions positions and drops the rest, so a caller
can take back tokens it fed, such as rejected speculative ones.
pub fn clear(&mut self)
Sourcepub fn append(
&mut self,
keys: &Matrix,
values: &Matrix,
) -> Result<(), NetworkError>
pub fn append( &mut self, keys: &Matrix, values: &Matrix, ) -> Result<(), NetworkError>
Appends one or more positions worth of keys and values.
pub fn keys(&self) -> &[f32]
pub fn values(&self) -> &[f32]
Trait Implementations§
impl StructuralPartialEq for KvCache
Auto Trait Implementations§
impl Freeze for KvCache
impl RefUnwindSafe for KvCache
impl Send for KvCache
impl Sync for KvCache
impl Unpin for KvCache
impl UnsafeUnpin for KvCache
impl UnwindSafe for KvCache
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more