Struct KvWindowPolicy 

pub struct KvWindowPolicy { /* private fields */ }

Whether this run may evict, decided once at load time.

A Copy value on the Decoder rather than a cached global, so a test can build a decoder that evicts and one that does not in the same process and compare their tokens. A global read once per process would need a subprocess per arm to say the same thing.

Implementations

impl KvWindowPolicy

pub const fn off() -> Self

Never evicts. What every cache in this engine did before #61, and what a run gets unless KV_WINDOW_ENV says otherwise.

pub const fn on() -> Self

Always evicts, where the layer has a window. For tests and for a caller that has already made the decision itself.

pub fn from_env() -> Self

Reads KV_WINDOW_ENV, then subtracts the runs that must not evict. See the module doc for which and why.

pub fn enabled(&self) -> bool

pub fn window_for(&self, attention_window: Option<usize>) -> Option<KvWindow>

The window a layer’s cache evicts behind, given the window that layer’s attention actually reads.

These two must be the same number. The kernel reads the last window rows; keeping fewer answers out of a truncated history, and this is the only place that says so, so there is nowhere for a second opinion to live.

pub fn layer_window( &self, config: &ModelConfig, layer_idx: usize, ) -> Option<KvWindow>

The window layer layer_idx of config evicts behind.

The ONE expression that answers it, for the decoder (Decoder::kv_window_for_layer) and for the budget (crate::kv_budget::KvResidency::from_config) alike. Those two have to agree about which layers keep how much or the budget is pricing a cache that does not exist, which is #33.

Trait Implementations

impl Clone for KvWindowPolicy

fn clone(&self) -> KvWindowPolicy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for KvWindowPolicy

impl Debug for KvWindowPolicy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for KvWindowPolicy

fn default() -> Self

Returns the “default value” for a type.

impl Eq for KvWindowPolicy

impl PartialEq for KvWindowPolicy

fn eq(&self, other: &KvWindowPolicy) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for KvWindowPolicy