Skip to main content

Module cache_breakpoint

Module cache_breakpoint 

Source
Expand description

Active prompt-cache breakpoint injection (#939, Headroom “cache aligner” adjacent).

Anthropic’s prompt cache is opt-in per request: the client marks a cache_control: {type:"ephemeral"} breakpoint and Anthropic caches the prefix up to it (billing later turns at the cached rate). A raw API client that does not set one pays full price for its (large, stable) system prompt on every single turn. This module injects exactly one breakpoint on the system field for those clients, so the proxy delivers the cache win the client left on the table.

§Anthropic-only by construction

cache_control is an Anthropic concept. OpenAI Chat Completions and the Responses API cache prefixes automatically (no per-request markers; an injected cache_control would be ignored at best), so there is nothing to inject there — those paths rely on OpenAI’s implicit caching and are left byte-unchanged. This module therefore wires into the Anthropic path only.

§Safety

  • Only when the client set none. The caller gates on cached_prefix_len(messages) == 0 and !prose::value_has_cache_control(system), so we never add a second breakpoint (Anthropic caps them at 4) or move a client anchor.
  • Exactly one, on system — the largest, most stable prefix, fixed at the very start, so it never churns with the prune boundary.
  • Deterministic (#498): a pure function of the body, so the rewritten request is byte-identical across turns and the cache prefix it creates is itself stable.
  • Min size. Below Anthropic’s minimum cacheable prefix the marker is ignored, so we skip tiny system prompts to avoid pointless churn.