Module metadata_cache 

Cache of per-launch kernel metadata info buffers. Cache of per-launch metadata info buffers (kernel shapes, strides and scalars) keyed by the exact info bytes they were built from (see [InfoCacheKey]).

Info depends only on shapes and scalar arguments — not on the tensor data pointers, which are separate kernel arguments — so two launches with identical info, even of different kernels, can share one read-only device buffer. Caching those buffers removes a fresh allocation and a host→device copy from every launch, and it is what makes hardware graph capture clean: a stable-shape decode’s launches all hit warm buffers, so nothing is allocated or copied inside the capture window (a device allocation mid-capture is illegal).

[MetadataCachePolicy] owns every decision: whether a given metadata info is worth caching at all (small enough, and the cache enabled) and how many entries to keep before the least-recently-used one is evicted. Its two knobs, max_entries and max_cached_size, are tuned by the backend. The current [CacheMode] feeds into those same decisions — during graph capture the policy caches everything and evicts nothing — so the runtime never has to special-case capture: it just asks the policy and follows the answer.

A captured graph records the device pointer of every info buffer its launches touch, so those buffers must outlive the graph. While a capture is recording, every entry the launch path touches — a fresh miss or a hit on a buffer built earlier in normal operation — is pinned to the graph being built. Pinned entries are never evicted, so the pointer a graph recorded stays valid for its whole life even if the cache would otherwise reclaim it. capture_commit seals those pins under the graph’s id at end_capture, and graph_release drops them when the graph is destroyed, freeing any buffer no other live graph still pins. Pins are refcounted, so an info buffer shared by two graphs survives until both are gone.

The runtime drives it in three steps, so a value is only ever cloned into the cache when it will actually be kept:

1. ask [MetadataInfoCache::should_cache] — if false, create the value and return it directly, never touching the cache;
2. otherwise get — a hit returns the cached value;
3. on a miss, create the value and insert it.

Structs

MetadataCachePolicy

Every caching decision, in one place. Given an info’s size and the current CacheMode, the policy decides whether that info is cached at all (should_cache) and, through its capacity, when the cache must evict to stay bounded. How those decisions are carried out (lookups, eviction) is the cache’s job and differs per runtime; what to do lives here.

MetadataInfoCache

A cache of metadata info buffers keyed by InfoCacheKey, generic over the value type V (a device buffer handle for a compute backend). Evicting an entry drops its V; when V is a memory handle that returns the buffer to the pool, so the cache never pins more device memory than its live entries.

Enums

CacheMode

How the cache should behave for the current launch. Fed into the MetadataCachePolicy, so it shapes every decision, not just a setting.

Type Aliases

InfoCacheKey

Identifies a cached info buffer: the exact metadata words (shapes, strides, scalars) the buffer was built from — nothing else.