Skip to main content

Module verified_cache

Module verified_cache 

Source
Expand description

A response cache that can only ever return an answer which passed a gate.

§Why this is not the same feature everyone else ships

A conventional semantic or exact cache stores whatever the model said and replays it. It saves money by skipping the model — and skipping the model is exactly how a gateway stops knowing whether the thing it just served was any good. On a cache hit rate of 40%, a product whose claim is “nothing ships until a real check passes” would be shipping 40% of its traffic unchecked.

So insertion is the gate here, not lookup: an entry is written only when the request it came from was served on a passing verdict. A failed answer, a best-effort serve after budget exhaustion, an abstain — none of them are cacheable, because replaying one would launder an unverified answer into a served one and no later reader could tell.

The receipt records served_from: cache and cache_source: <trace_id>, so a hit is not the word “cached” but a link to the decision whose gate passed. That is the difference between asserting a saving and being able to prove what was served.

§What it is keyed on

The salted prompt hash the router already computes, plus the ladder identity. Exact-match, not semantic: two prompts that merely look similar can want different answers, and a verification product guessing that they don’t is the one shortcut it cannot take. Semantic matching would need its own gate over the similarity decision, which is a bigger design than a cache.

The ladder is part of the key because the same prompt under a different ladder is a different question — an answer proven against one gate configuration says nothing about another.

Structs§

Entry
An answer that passed, kept for replay.
VerifiedCache
A bounded, in-process store of verified answers.