Struct BackendCache 

pub struct BackendCache<T: Send + 'static> { /* private fields */ }

LRU-bounded cache of loaded inference backends.

Not itself Sync over T — the cache stores handles only, and the lock guarding the map is coarse (held only for look-up / insert / evict). Per-entry mutation is serialized by each entry’s own Mutex.

Implementations

impl<T: Send + 'static> BackendCache<T>

pub fn new(budget_bytes: u64) -> Self

Create a new cache with the given RAM budget in bytes.

pub fn from_env() -> Self

Read the CAR_INFERENCE_MODEL_CACHE_MB env var, default 24 GB. A value of 0 disables caching (every call loads fresh).

pub fn is_disabled(&self) -> bool

Is this a disabled (zero-budget) cache?

pub fn get_or_load<E>( &self, key: &str, size_bytes: u64, loader: impl FnOnce() -> Result<T, E>, ) -> Result<CachedBackend<T>, E>

Get a handle to the cached backend, or load + insert it.

loader is invoked only on a cache miss. If the new entry plus existing entries exceed the budget, the oldest entries are evicted until we’re within budget (or only the new entry remains).

size_bytes should be an approximate on-disk or in-memory size for the loaded backend; estimate_model_size is a reasonable default that sums the .safetensors file sizes in a model dir.

pub fn invalidate(&self, key: &str)

Manually remove a key, e.g. when model weights have been updated on disk. Outstanding Arc<Mutex<T>> handles remain valid.

pub fn clear(&self)

Evict all entries. Outstanding handles continue to work.

pub fn stats(&self) -> (usize, u64, u64)

Returns (entries, total_bytes, budget_bytes) for diagnostics.