Struct CacheRuntime 

pub struct CacheRuntime { /* private fields */ }

Ties policy resolution + content-addressed key derivation + single-flight compute-through into one call the dispatch path makes per tool. The enterprise injects a Redis backend + the real tenant; the OSS default is an in-process backend under a "local" tenant. This is the whole runtime contract for v2.40.0 — a hit returns before compute runs (so a budget gate placed after the lookup never sees it, the design decision).

Implementations

impl CacheRuntime

pub fn new(backend: Arc<dyn CacheBackend>, tenant: impl Into<String>) -> Self

pub fn in_process() -> Self

In-process, single-tenant default (OSS runtime with no injected tier).

pub fn process_local(tenant: impl Into<String>) -> Self

v2.89.0 — the OSS runtime a production flow run gets: the process-wide in-process tier, keyed to this run’s tenant.

Why the split, and what each half prevents

This looks like a detail and is the difference between a cache and a decoration.

Build the whole CacheRuntime per flow run and the backend is empty every time: a tool called once per run — which is most tools — never hits anything, and v2.89.0 would ship a memoiser that memoises within a single run and forgets between them. Wired, tested, and worthless.

Share the whole CacheRuntime across runs and it is worse than worthless. tenant is a field of the runtime, not a parameter of the call, so one shared instance would key every tenant’s results under whichever tenant built it first — and the design decision puts the tenant IN the key precisely so that a mis-namespacing backend still cannot leak. A shared runtime with a fixed tenant defeats that from above the backend, where the key is derived.

So the BACKEND is process-wide (entries survive between runs, which is what makes it a cache) and the TENANT comes from the run (which is what keeps them apart). Constructing this is two Arc clones.

The enterprise v2.40.0 Redis tier replaces the backend here and inherits the same discipline unchanged — it is a different CacheBackend, not a different call site.

pub fn dispatch<F, E>( &self, ir: &IRProgram, tool: &IRToolSpec, args: &[(String, String)], compute: F, ) -> Result<CacheOutcome, E>

where F: FnOnce() -> Result<Vec<u8>, E>,

Look up (or compute-and-store) a tool result. args is the full bound (name, value) set; the key: subset (if any) is applied here. compute runs ONLY on a miss and its error is never cached.

pub fn dispatch_resolved<F, E>( &self, policy: Option<&ResolvedCachePolicy>, subject: &str, args: &[(String, String)], compute: F, ) -> Result<CacheOutcome, E>

where F: FnOnce() -> Result<Vec<u8>, E>,

v2.89.0 — the memoisation body, against an already-resolved policy.

subject is what the entry is keyed to — a tool’s name, or a store’s name for a retrieve. policy: None means “nothing memoises this”: compute runs and the value comes back as CacheOutcome::Uncached, which the caller uses exactly as a Miss minus the audit signal.

The ordering that the design decision rests on

A hit returns BEFORE compute is called. That is not an optimisation, it is the guarantee: the caller places its budget charge inside compute’s caller, so a hit cannot decrement a budget { rate: … } quota. v2.40.0’s plan calls this “structurally guaranteed by ordering the cache lookup before the budget gate” — the structure is right here.

pub fn probe( &self, policy: Option<&ResolvedCachePolicy>, subject: &str, args: &[(String, String)], ) -> CacheProbe

v2.89.0 — look, without computing.

Why the seam had to split

dispatch_resolved takes a compute closure, which is the right shape when the work is synchronous and owns nothing. The real tool-call path is neither: the work between the lookup and the value is async, it borrows &mut DispatchCtx, and it is not one call but four in sequence — the budget charge, the lease charge, the concurrency permit, then the vendor dispatch. None of that fits inside an FnOnce() -> Result<Vec<u8>, E>, and contorting it to fit would have meant either blocking the executor or duplicating the memoisation law at the call site.

So the law splits into the two moments an async caller actually has: probe before the work, store after it. dispatch_resolved is now implemented in terms of these, so the synchronous seam v2.40.0 designed and the asynchronous one v2.89.0 needed run the SAME key derivation, the same TTL parse and the same namespace — one law, two ways in.

The ordering the design decision rests on is the caller’s to keep: a CacheProbe::Hit means the call must not be dispatched AND no budget charged. Returning early on a hit is what makes “a hit never consumes a quota” structural rather than hopeful, and it is asserted from a real deploy in cache_hits.rs.

pub fn store(&self, slot: &CacheSlot, value: Vec<u8>)

v2.89.0 — fill the slot a CacheProbe::Miss reserved.

Errors are never stored — that is the caller’s decision, and it is expressed by simply not calling this. Oversized values are dropped by the backend, never truncated into a wrong value.

pub fn invalidate(&self, cache_name: &str)

Flush a cache namespace (called when an emit fires on one of its invalidate_on: channels).