Module cold_prefix 

Big-gap cold-prefix repack prediction (#480).

The proxy is deliberately cache-safe: it never rewrites the client-cached prefix (history_prune::cached_prefix_len), so provider prompt caches keep hitting and cheap cache reads (~0.1x) never turn into full-price writes (~1.25x) — the #448 invariant.

That protection has one blind spot. Provider prompt caches EXPIRE after a TTL of inactivity. After a long idle gap (the agent asked a question, the user replies hours later) the cached prefix is already gone; the provider will re-WRITE the whole prefix on the next request regardless. Staying in “never touch the cached prefix” mode then writes the uncompressed prefix at full price and re-seeds a fat cache for the rest of the session.

This module makes a PRE-SEND prediction — purely from elapsed idle time vs the provider’s cache TTL — of whether the prefix is already cold. The trigger must be a clock, not response feedback: hit/miss is only known after the request that already (re-)cached the prefix, and by the next request the cache is warm again, so feedback would bust the fresh cache.

Safety is paramount because the cost of a wrong “cold” guess is asymmetric (a cache write is ~12x a cache read). We therefore:

• act only when the caller opted in (repacks_cold_prefix()),
• act only on a measured idle gap well past expiry (TTL × SAFETY_MARGIN, with an absolute floor), skipping the ambiguous near-TTL zone entirely,
• never act without a prior touch (the first sighting only sets a baseline),
• and bias every ambiguity toward “warm” (do nothing).

State persists across restarts ({data_dir}/cold_prefix_touch.json, atomic write, throttled) so an idle gap that straddles a daemon recycle is still detected — a stale on-disk timestamp is exactly what proves the gap and can only ever bias toward “warm” if lost (#499). A missing/corrupt file simply disables the optimization until a fresh baseline is recorded — a safe degradation that can never wrongly trigger.

Once a conversation is judged cold and repacked, the decision is sticky: every later turn keeps applying the same deterministic prefix compression, so the warm follow-ups that resume active use hit the compressed prefix written at the cold turn instead of re-sending the uncompressed original and busting the freshly-seeded cache (#499). Deterministic re-compression is prefix- stable, so the latch stays cache-safe for the rest of the session.

Functions

repack_decision

Decide whether to repack the (predicted-cold) cached prefix for THIS request, recording this request as the conversation’s latest touch.

resume_from_disk

Seeds the in-memory baselines from disk on proxy startup so an idle gap that straddles a restart is still detected. Merges by most-recent last_touch and OR-s the sticky repacking latch, so a re-seed can only bias toward “warm” (or keep a latch), never toward a wrong “cold”.