Struct KvCacheParams 

#[non_exhaustive]
pub struct KvCacheParams {
    pub type_k: KvCacheType,
    pub type_v: KvCacheType,
}

KV cache quantization configuration.

Controls the data type used for the attention K and V caches. llama.cpp defaults both to F16 (GGML_TYPE_F16), which is what KvCacheParams::default() preserves. Quantizing the KV cache (e.g. Q8_0 → ~½ size, Q4_0 → ~¼ size) trades a small amount of accuracy for a large reduction in VRAM usage, which is often the dominant cost at long n_ctx.

Marked #[non_exhaustive]; build via Default::default() and chain the with_* setters:

use rig_llama_cpp::{KvCacheParams, KvCacheType};

let kv = KvCacheParams::default()
    .with_type_k(KvCacheType::Q8_0)
    .with_type_v(KvCacheType::Q8_0);

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

type_k: KvCacheType

Data type for the K cache (default: KvCacheType::F16).

type_v: KvCacheType

Data type for the V cache (default: KvCacheType::F16).

Implementations

impl KvCacheParams

pub fn with_type_k(self, type_k: KvCacheType) -> Self

Override the K cache data type.

pub fn with_type_v(self, type_v: KvCacheType) -> Self

Override the V cache data type.

Trait Implementations

impl Clone for KvCacheParams

fn clone(&self) -> KvCacheParams

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KvCacheParams

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for KvCacheParams

fn default() -> Self

Returns the “default value” for a type.

impl Copy for KvCacheParams