Enum CacheStateStrategy 

pub enum CacheStateStrategy {
    FullKv,
    SlidingKv {
        window: u64,
    },
    SlidingKey {
        window: u64,
        layers: u64,
        pooling_layers: u64,
    },
    MixedKv {
        full_layers: u64,
        sliding: Vec<SlidingWindowLayerCount>,
    },
    SharedFullKv {
        cached_layers: u64,
        shared_layers: u64,
        full_attention_layers: u64,
        sliding_attention: Vec<SlidingWindowLayerCount>,
    },
    CompressedMla {
        latent_width: u64,
        rotary_width: u64,
    },
    HybridRecurrent {
        full_attention_layers: u64,
        sliding_attention: Vec<SlidingWindowLayerCount>,
        recurrent_layers: u64,
    },
    Multimodal {
        decoder: Box<CacheStateStrategy>,
        media_consumes_decoder_positions: bool,
    },
}

Persistent decoder-state strategy used by a model.

Variants

FullKv

Ordinary full-context K/V attention.

SlidingKv

Every attention cache is bounded by a sliding window.

Fields

window: u64

Maximum retained positions per attention layer.

SlidingKey

Sliding-window key-only attention, optionally with pooled history.

Fields

window: u64

Maximum retained key positions per attention layer.

layers: u64

Total layers retaining local keys.

pooling_layers: u64

Layers that additionally retain append-only pooling state.

MixedKv

Full-context and sliding-window attention layers.

Fields

full_layers: u64

Number of full-context layers.

sliding: Vec<SlidingWindowLayerCount>

Bounded layer counts grouped by retained window.

SharedFullKv

Full-context KV backing with layers that reuse earlier K/V state.

Fields

cached_layers: u64

Layers that allocate their own K/V state.

shared_layers: u64

Layers that reuse K/V produced by an earlier layer.

full_attention_layers: u64

Total full-attention layer count.

sliding_attention: Vec<SlidingWindowLayerCount>

Sliding-mask layers, which do not bound KV allocation.

CompressedMla

Multi-head latent attention compressed state.

Fields

latent_width: u64

Compressed latent width per layer and position.

rotary_width: u64

Shared rotary-key width per layer and position.

HybridRecurrent

Attention combined with bounded convolution or recurrent state.

Fields

full_attention_layers: u64

Full-context attention layer count.

sliding_attention: Vec<SlidingWindowLayerCount>

Bounded attention layers grouped by exact window.

recurrent_layers: u64

Recurrent/linear-attention layer count.

Multimodal

Multimodal preparation feeding positions into a decoder strategy.

Fields

decoder: Box<CacheStateStrategy>

Underlying decoder state.

media_consumes_decoder_positions: bool

Whether media embeddings consume persistent decoder positions.

Trait Implementations

impl Clone for CacheStateStrategy

fn clone(&self) -> CacheStateStrategy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for CacheStateStrategy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for CacheStateStrategy

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for CacheStateStrategy

impl PartialEq for CacheStateStrategy

fn eq(&self, other: &CacheStateStrategy) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for CacheStateStrategy

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for CacheStateStrategy