Skip to main content

Module cache_aligner

Module cache_aligner 

Source
Expand description

Cache-aligner (#940 detect, #974 relocate) — Headroom “cache aligner” port.

A stable system prompt is the largest prefix a provider can cache, but a single turn-to-turn-varying token inside it (today’s date, a fresh UUID, a git SHA) shifts the bytes and busts the cache on every request. Two opt-in stages address this, both Anthropic-only:

  1. Detect (cache_aligner, #940): a deterministic scan counts the volatile fields in an unanchored system prompt and surfaces the leak on /status — pure measurement, the body is never mutated.
  2. Relocate (cache_align_relocate, #974): rewrites system into a stable block (volatile values replaced by constant placeholders) carrying the cache breakpoint, plus an uncached tail block that re-states the relocated values. The cacheable prefix then stays byte-stable turn-to-turn and finally caches; only the small, reprocessed tail changes. Follows the same stable-first ordering as crate::core::neural::cache_alignment::CacheAlignedOutput.

§Determinism (#498) & cache-safety (#448)

Both stages are pure functions of the text: matches come from the merged_spans helper (collected, sorted, overlaps merged), and the relocate’s placeholders + tail header are byte-constants, so identical input yields byte-identical output and the rewritten prefix is stable across turns. The relocate is idempotent (a second pass sees only placeholders) and only ever fires when the client anchored nothing itself, so it never rewrites a client-cached prefix.