Enum OffloadPolicy 

pub enum OffloadPolicy {
    None,
    Budget {
        ram_bytes: u64,
    },
    PinnedHotSet {
        ram_bytes: u64,
        pin_embeddings: bool,
        pin_output_head: bool,
        pin_last_n_layers: usize,
        prefetch_n_ahead: usize,
    },
}

Configures which weights stay in RAM vs. get evicted to disk.

Variants

• None — all layer weights remain in RAM. This is the default and matches the classic llama.cpp in-memory-only behaviour.
• Budget — evict weights until the resident set fits within ram_bytes. The LRU pager is activated; any tensor not pinned can be evicted.
• PinnedHotSet — like Budget but a named hot-set (embeddings, output head, last N attention layers) is pinned and never evicted. Cold layers cycle in and out of RAM as they are needed.

Variants

None

All layer weights remain in RAM (default, current behaviour).

Budget

Evict weights until resident set fits within ram_bytes.

Fields

ram_bytes: u64

Maximum number of bytes allowed to be resident simultaneously.

PinnedHotSet

Pinned hot-set: evict cold layers but keep embeddings, output head, and the last N attention layers always resident.

Fields

ram_bytes: u64

Maximum number of bytes allowed to be resident simultaneously.

pin_embeddings: bool

Keep token-embedding table pinned (never evicted).

pin_output_head: bool

Keep output LM head pinned (never evicted).

pin_last_n_layers: usize

Number of trailing (deepest) attention layers to always keep resident.

prefetch_n_ahead: usize

How many layers ahead to prefetch in the background.

Currently advisory — the pager does not spawn a prefetch thread automatically; callers can use this value to decide whether to pre-acquire tensors ahead of the current layer index.

Implementations

impl OffloadPolicy

pub fn ram_budget_bytes(&self) -> Option<u64>

Return the RAM budget in bytes, if any eviction limit is set.

Returns None for OffloadPolicy::None (unlimited RAM usage).

pub fn is_disabled(&self) -> bool

Returns true if offloading is disabled (i.e. the default in-RAM path).

Trait Implementations

impl Clone for OffloadPolicy

fn clone(&self) -> OffloadPolicy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for OffloadPolicy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for OffloadPolicy

fn default() -> OffloadPolicy

Returns the “default value” for a type.