Struct CompileCache 

pub struct CompileCache { /* private fields */ }

Implementations

impl CompileCache

pub fn new(device: Device, capacity: usize) -> Self

pub fn with_policy( device: Device, capacity: usize, policy: Option<PrecisionPolicy>, ) -> Self

Cache that compiles every entry with the given precision policy. Use this when the cached entries should differ from CPU-default F32 — e.g., PrecisionPolicy::AutoMixed for f16 compute on Metal.

pub fn get_or_compile<F: FnOnce() -> Graph>( &mut self, key: u64, build: F, ) -> &mut CompiledGraph

Compile if not present, then return a mutable reference. The borrow lifetime is tied to &mut self so callers naturally serialize their use of any one entry — the cache is single-owner today.

pub fn get_or_compile_with_options<F: FnOnce() -> Graph>( &mut self, key: u64, build: F, options: &CompileOptions, ) -> &mut CompiledGraph

Like Self::get_or_compile with explicit CompileOptions.

pub fn ensure_with_params( &mut self, key: u64, graph: Graph, params: &HashMap<String, Vec<f32>>, options: &CompileOptions, ) -> &mut CompiledGraph

Compile key if absent, then give it weights — retaining another entry’s uploaded buffer when the layout matches, and uploading params only when it does not.

The point of the cache is holding several shapes of one model at once, so without sharing its own hit rate is what costs the memory: each extra prompt length is another full copy of the weights, host and device.

pub fn try_share_params_from_donor(&mut self, dst_key: u64) -> bool

Retain another entry’s weight buffer for dst_key. False when there is no compatible entry and the caller must upload.

pub fn get_or_compile_hir_with_options<F: FnOnce() -> HirModule>( &mut self, key: u64, build: F, options: &CompileOptions, ) -> &mut CompiledGraph

Like Self::get_or_compile_with_options but builds and compiles HIR directly.

pub fn len(&self) -> usize

Number of entries currently cached. Useful for tests + diagnostics.

pub fn is_empty(&self) -> bool

pub fn contains(&self, key: u64) -> bool

Was this key already compiled? Doesn’t change recency.

pub fn clear(&mut self)

Drop all cached compiled graphs (free weight params).

pub fn sync_all(&mut self)

Drain in-flight GPU work on every cached entry (Metal commit_no_wait paths).