Expand description
A response cache that can only ever return an answer which passed a gate.
§Why this is not the same feature everyone else ships
A conventional semantic or exact cache stores whatever the model said and replays it. It saves money by skipping the model — and skipping the model is exactly how a gateway stops knowing whether the thing it just served was any good. On a cache hit rate of 40%, a product whose claim is “nothing ships until a real check passes” would be shipping 40% of its traffic unchecked.
So insertion is the gate here, not lookup: an entry is written only when the request it came from was served on a passing verdict. A failed answer, a best-effort serve after budget exhaustion, an abstain — none of them are cacheable, because replaying one would launder an unverified answer into a served one and no later reader could tell.
The receipt records served_from: cache and cache_source: <trace_id>, so a hit is not the
word “cached” but a link to the decision whose gate passed. That is the difference between
asserting a saving and being able to prove what was served.
§What it is keyed on
The salted prompt hash the router already computes, plus the ladder identity. Exact-match, not semantic: two prompts that merely look similar can want different answers, and a verification product guessing that they don’t is the one shortcut it cannot take. Semantic matching would need its own gate over the similarity decision, which is a bigger design than a cache.
The ladder is part of the key because the same prompt under a different ladder is a different question — an answer proven against one gate configuration says nothing about another.
Structs§
- Entry
- An answer that passed, kept for replay.
- Verified
Cache - A bounded, in-process store of verified answers.