pub enum KvMode {
F32,
Q8 {
k: bool,
v: bool,
},
}Expand description
KV storage mode. CMF_KV=q8 enables the q8_2f cache: an int8 row per
(position, head) + an f32 scale per row + a per-channel field ๐ฒรฮธ,
frozen after WARMUP positions with retroactive requantization
(D4: โKV-quant 2fโ). Memory ร~3.7 smaller than f32.
Variantsยง
Implementationsยง
Trait Implementationsยง
impl Copy for KvMode
impl Eq for KvMode
impl StructuralPartialEq for KvMode
Auto Trait Implementationsยง
impl Freeze for KvMode
impl RefUnwindSafe for KvMode
impl Send for KvMode
impl Sync for KvMode
impl Unpin for KvMode
impl UnsafeUnpin for KvMode
impl UnwindSafe for KvMode
Blanket Implementationsยง
Sourceยงimpl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Sourceยงfn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more