Struct KernelCache 

pub struct KernelCache { /* private fields */ }

Thread-safe LRU cache of compiled kernels backed by dashmap::DashMap.

get and put are O(1) and (under typical concurrent workloads) lock- free except for the per-shard dashmap lock and the eviction-policy parking_lot::Mutex. The eviction lock is taken only on puts that would push the cache over capacity; reads never touch it.

Implementations

impl KernelCache

pub fn new() -> Self

Construct with default capacity.

pub fn with_capacity(cap: usize) -> Self

Construct with explicit capacity. Anything below 1 is clamped to 1.

pub fn with_config(config: KernelCacheConfig) -> Self

Construct from a full KernelCacheConfig. Anything below 1 in config.capacity is clamped to 1.

pub fn with_disk_persistence(self, cfg: DiskCacheConfig) -> Self

Enable the on-disk L2 cache, persisting entries to cfg.dir under an HMAC-keyed integrity tag (jit S-3).

Construct via:

use std::path::PathBuf;
use tensor_wasm_jit::cache::{KernelCache, DiskCacheConfig};
let cache = KernelCache::new().with_disk_persistence(DiskCacheConfig {
    dir: PathBuf::from("/var/cache/tensor-wasm/kernels"),
    hmac_key: load_hmac_key_from_secret_store()?, /* loaded from secrets at startup */
});

jit S-3 (T13): the example deliberately routes the key through a load_hmac_key_from_secret_store() stub rather than an inline literal — operators have a habit of copy-pasting rustdoc examples verbatim, and an embedded [0xAB; 32] (or similar) would survive into production deployments as a fixed, attacker-known key.

The directory is created lazily on the first put. The HMAC key MUST be stable across process restarts AND treated as a server- side secret — possession of the key lets an attacker forge cache entries that the loader will accept as authentic (and that would then be handed to cust::module::Module::from_ptx as trusted GPU code).

pub fn put(&self, key: CacheKey, kernel: CachedKernel)

Insert (or replace) a kernel. If the insert pushes the cache over the count cap (or, when configured, the byte cap), evicts LRU entries from storage and the policy queue until both caps are satisfied.

jit S-3: the kernel’s integrity_hash is recomputed and compared against the stored hash; a mismatch is treated as a programmer error (CachedKernel constructed via struct-literal with a wrong hash), logged at error!, and the entry is dropped rather than admitted. Use CachedKernel::new for the correct-by- construction path. The on-disk L2 also writes the entry when configured.

pub fn get(&self, key: &CacheKey) -> Option<Arc<CachedKernel>>

Look up a kernel; best-effort touches the LRU position.

T20 perf: returns Option<Arc<CachedKernel>> rather than the previous Option<CachedKernel> so a cache hit shares the wrapper allocation via refcount bump instead of cloning the 32-byte integrity hash + fingerprint + inner-Arc refcount bump. The inner PTX text was already shared; this closes the matching gap on the outer wrapper.

pub fn get_for( &self, tenant_id: TenantId, blueprint: &TensorWasmKernelBlueprint, sm_version: u32, ) -> Option<Arc<CachedKernel>>

Look up by blueprint + sm_version for a given tenant; convenience wrapper around Self::get.

T20 perf: returns Option<Arc<CachedKernel>> to mirror Self::get; callers that only need to peek at fields go through auto-deref unchanged.

pub fn get_with_registry_fallback( &self, key: &CacheKey, resolver: &dyn BlueprintResolver, ) -> Option<Arc<CachedKernel>>

L1 → L2 → L3(registry) resolution path. v0.3.8 scaffold: invokes the registered resolver + registry on every miss; v0.4 may add a resolver-level cache to amortise the (blueprint → name@version) translation.

pub fn len(&self) -> usize

Number of entries currently held.

pub fn is_empty(&self) -> bool

True if empty.

pub fn capacity(&self) -> usize

Configured capacity.

pub fn total_bytes(&self) -> u64

Running sum of resident PTX-text bytes across all L1 entries (each entry contributes cached.ptx.text.len()). This is the quantity the optional KernelCacheConfig::max_total_bytes cap bounds. Surface on the Prometheus gauge tensor_wasm_jit_cache_bytes so operators can watch the L1 footprint against the configured byte cap. Always present (returns 0 for an empty cache, and tracks the live total whether or not a byte cap is configured).

pub fn max_total_bytes(&self) -> Option<u64>

The configured byte cap, if any (mirror of KernelCacheConfig::max_total_bytes). None means count-only eviction. Useful for diagnostic endpoints that surface both the live Self::total_bytes and the ceiling it is measured against.

pub fn active_disk_key_fingerprint(&self) -> Option<KeyFingerprint>

Fingerprint of the active on-disk HMAC key — the partition prefix every file this cache writes lands under — or None if no L2 disk cache is configured.

Pass this into the retain set of Self::gc_disk to keep the live generation (though gc_disk retains it unconditionally as a safety net).

pub fn disk_key_fingerprints(&self) -> Result<Vec<KeyFingerprint>>

Enumerate the distinct HMAC-key fingerprints present in the on-disk L2 cache directory (one per key generation that has written at least one file). Returns an empty Vec when no disk cache is configured or the directory does not yet exist.

Use this to discover stale generations before a Self::gc_disk sweep — anything in this list that is not the active fingerprint and not a key you still want to honour is a rotation leftover.

pub fn gc_disk(&self, retain: &[KeyFingerprint]) -> Result<usize>

Garbage-collect the on-disk L2 cache after an HMAC-key rotation: remove every persisted entry whose key fingerprint is NOT in retain, returning the number of files removed. A no-op returning Ok(0) when no disk cache is configured.

The active key’s fingerprint is always retained — even if the caller omits it from retain — so a sweep can never delete the live generation’s entries. Only stale-fingerprint files are removed, and only files matching the cache’s own naming scheme are considered (unrelated files in the directory are left untouched).

Example

use std::path::PathBuf;
use tensor_wasm_jit::cache::{KernelCache, DiskCacheConfig};

// After rotating to a fresh HMAC key, stand up a cache under it…
let cache = KernelCache::new().with_disk_persistence(DiskCacheConfig {
    dir: PathBuf::from("/var/cache/tensor-wasm/kernels"),
    hmac_key: load_hmac_key_from_secret_store()?,
});

// …then sweep every previous generation's files. Retaining only the
// active key (which `gc_disk` keeps regardless) drops all the rest.
let active = cache.active_disk_key_fingerprint().into_iter().collect::<Vec<_>>();
let removed = cache.gc_disk(&active)?;
println!("swept {removed} stale-fingerprint cache files");

pub fn config(&self) -> &KernelCacheConfig

Borrow the construction-time KernelCacheConfig. Useful for tests and diagnostic endpoints that want to surface whether verify_on_get is on for this cache instance.

pub fn verify_skipped_total(&self) -> u64

Cumulative count of L1 get hits that skipped the BLAKE3 integrity recompute because KernelCacheConfig::verify_on_get is false. Surface this on the Prometheus counter tensor_wasm_jit_cache_verify_skipped_total so operators can see how often the cache is trusting an L1 entry without re-hashing the PTX. The counter is always present (returns 0 when verify_on_get is on) so dashboards can scrape it unconditionally.

pub fn cache_hits_total(&self) -> u64

Cumulative L1/L2/L3 cache hit count. Incremented once per get call that returns Some. Surface on the Prometheus counter tensor_wasm_jit_cache_hits_total so operators can compute the hit ratio against Self::cache_misses_total. (T20 perf.)

pub fn cache_misses_total(&self) -> u64

Cumulative cache miss count. Incremented once per get call that returns None — including the rare path where an L1 entry failed integrity verification and was evicted. Surface on the Prometheus counter tensor_wasm_jit_cache_misses_total. (T20 perf.)

pub fn integrity_reject_total(&self) -> u64

Cumulative count of entries refused on get because they failed integrity verification — an L1 BLAKE3 recompute mismatch, an all-zero integrity_hash on the verify-skip path, or an L2 disk integrity failure (artifact-store HMAC/content-hash mismatch, inner-envelope magic/header/length/UTF-8 corruption). These also bump Self::cache_misses_total, but this dedicated counter lets operators alarm on tamper attempts specifically rather than drown them in ordinary cold-cache misses. Surface on the Prometheus counter tensor_wasm_jit_cache_integrity_reject_total.

Trait Implementations

impl Clone for KernelCache

fn clone(&self) -> KernelCache

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for KernelCache

fn default() -> Self

Returns the “default value” for a type.