Module tool_keepwarm 

Keep the prompt cache warm while a provider round runs (cache_keepwarm_tools).

A provider round that asks for tools is followed, once they return, by a request that re-sends that round’s whole prefix. The cache entry that request reads lives for one TTL from the START of the round’s request (workers are pinned to 5m), and a round can outlast it twice over:

• while its response streams. One long generation is enough: minutes of thinking before a short tool call, or a 30k-token write.
• while its tools run. Builds, test sweeps and sleep; poll loops routinely take that long.

Either way the entry has expired by the time the next request goes out, and that request rewrites the whole prefix at the write price (1.25× on 5m) instead of reading it (0.1×, 0.05× on the 5.5 models).

From the moment a round’s request is sent, this task re-sends that round’s own request — the exact request messages, tools, system prompt, thinking config and options it went out with — plus one trailing ping text block (KeepWarmMode::Replay, appended by the transport after marker placement), shortly before the TTL lapses, and again after every accepted ping, until the tools return, the response ends without asking for any, the turn is cancelled or the stream ends (the guard drops). It is a keep-warm ping in every other respect (runtime::keepwarm): ApiOptions::keepwarm cuts max_tokens, disables retries and marks the telemetry record "keepwarm": true; the transport holds the ping text out of marker placement, so every breakpoint lands on the block the round itself marked and the ping is an uncached tail. A read refreshes the entry’s TTL, so the request after the round reads the prefix the round cached. Nothing is persisted and no history is touched.

Policy (the idle keep-warm’s, session::cache_keepwarm, minus most of the idle guesswork). While the tools run, the next request is certain unless the turn is cancelled. While the response streams it is not: the response may be the turn’s last word. But a response that outlasts the TTL has lost its cache by the time it ends, whatever reads the prefix next (its tool results, a resume of the worker, the user’s next message), and what a long response does first is think: the tool call comes at the end, too late to wait for. So the response is pinged as the tools are, under the same bound:

• eligible only on the native Anthropic transport and when the round’s request cached something. While the response streams that is the prefix the provider reports in message_start, passed on by the transport (RoundSignal): the round’s own usage is not booked until the response ends;
• due at anchor + ping_interval(honoured TTL) (4.5 min on 5m, 55 min on 1h) on whichever clock comes first; never once the cache is within LATE_GUARD of expiry (it is presumed gone: the next request rewrites it anyway, a ping would only add a write);
• ski-rental bound: a ping is sent only while ledger + expected ping ≤ rewriting the prefix at the honoured TTL;
• the turn’s max_cost_usd (if any) must admit the ping’s worst case. While the response streams, the round’s own request is not booked yet: it is counted at the most its input side can cost, a full write of the prefix;
• stop on a ping that read nothing, wrote or paid uncached for more than SHARE_LIMIT_PERCENT of the prefix, a 4xx, or exhausted transient retries (BACKOFF).

Stopping. When the tools return, no new ping is scheduled, but a ping already in flight is allowed to finish — for up to FINISH_GRACE — and is booked like any other before the next request goes out. Only a turn cancel, the stream ending, or the grace running out aborts one in flight. Either way the ping is settled before the round’s next request is sent, so its usage never lands after that request’s. (Provider-side order would not matter anyway: an aborted ping and the next request read the same prefix, and a ping writes nothing — its tail is uncached.) A response that ends without asking for tools ends the round and its pings: one in flight is aborted and booked before the turn goes on. (A ping it got while streaming is spent on a prefix that only a resume or the next turn can still read; the idle keep-warm, where one runs, schedules from the anchor that ping left.)

Booking. Every ping that may have been billed is booked into the turn, by the idle keep-warm’s rule (keepwarm::ping_booking): with measured usage the transport records it in the turn’s usage counters (turn budget) and its Usage event is forwarded to the turn’s event channel (session / worker cost); a ping dispatched with no usage seen (aborted in flight, a read timeout) is booked at its worst case — the whole prefix read, PING_INPUT_ESTIMATE uncached, max_tokens out — through both; one refused before the network or rejected with an HTTP error books nothing. A ping sent while the response streams is booked when it completes, normally before the round’s own usage; both are sums, so the order does not matter to them. UsageCounters::latest_context_tokens is the one last-writer field: a ping that completes in the instant between the round’s own booking and the stream loop’s read leaves its own footprint there, which is the round’s request plus the ping text (~40 tokens).

Structs

RoundSignal

What a round’s own request tells its keep-warm while the response is still streaming ([ApiOptions::round_signal]): written by the transport as the response starts, and by the stream loop once it is in; read by the round’s task. One per round, never shared with a ping (a ping’s own message_start must not pass for the round’s).

ToolKeepWarm

Per-turn settings: present only when cache_keepwarm_tools is on.

Constants

FINISH_GRACE

How long the end of a tool round waits for a ping already in flight before aborting it (and booking it at its worst case). Longer than every keep-warm ping in the api-log so far (122 real pings: p50 1.9 s, p95 11 s, max 15.8 s): the wait only happens when a ping straddles the end of the round (a few seconds out of every 270 s interval), and it delays the next request by at most the rest of that ping.

Functions

interval_override

Test seam replacing ping_interval(honoured TTL) for tool-round pings. Separate from the idle keep-warm’s seam so tests of one never fire the other. A constant None without cfg(test) / the testing feature; no binary input (env, config, argv) can reach it.