pub fn outer_pull_deadline(per_candidate: Duration, stall: Duration) -> DurationExpand description
Outer deadline for a node-to-node pull-through, derived from the two configured
per-candidate budgets: the stream-open timeout (cache.node_pull_timeout_sec) and
the streaming throughput-floor window (cache.node_pull_stall_window_sec).
The delivery handler wraps the whole discover → probe → rank → pull fetch
in a single tokio::time::timeout. For the sequential MAX_PROVIDER_ATTEMPTS
fallback loop to actually reach candidates #2..N when candidate #1 stalls,
this outer deadline must strictly exceed the sum of all per-candidate costs —
otherwise both clocks expire
together and the outer timeout cancels the whole fetch at the instant candidate
#1’s own timeout fires, killing the fallback.
§What one candidate actually costs
THREE sequential bounded stages:
- the channel open —
CHANNEL_OPEN_CALLER_BUDGET(#1143); then - the stream open — connect → handshake → verified
StreamResponse, bounded byper_candidate(#1134); then - streaming, which for a peer that opens honestly and then goes SILENT costs
one full throughput-floor window (
stall) before the pull gives up on it (#1797).
So the worst case for a candidate is CHANNEL_OPEN_CALLER_BUDGET + per_candidate + stall,
and that is what this must budget MAX_PROVIDER_ATTEMPTS of, plus
PULL_THROUGH_OUTER_SLACK of one-time discovery overhead.
Every stage has to be a term here. Budgeting only per_candidate would let a cold
cache against a slow L2 burn the whole deadline on candidates #1 and #2 and never
dial #3. Covering the channel open but not the stall window would leave the same hole
for a peer that goes silent mid-stream instead of failing to open — and a worse one,
because stall is operator-tunable: at node_pull_stall_window_sec = 120 a single
silent candidate outlasts a two-term deadline on its own. Taking stall as an
argument keeps this deadline in step with node_pull_stall_window_sec.
§What this deadline does not bound
A candidate’s streaming stage is bounded by inactivity rather than a wall clock,
because a wall clock over the bytes caps the blob size a node can pull through
(#1134). The stall term above is therefore the cost of a candidate that stops
— not a ceiling on one that keeps going. A candidate that is slow but progressing
resets that clock on every byte and can consume the whole outer deadline on its own.
That is correct: it is succeeding, and falling through mid-stream would restart the download from zero against another peer, throwing away the bytes already paid for. On expiry the foreground request gives up with a clean miss and nothing continues in the background — there is no detached warm, so a pull only ever runs while a client is waiting, and the blob is re-acquired on the next real client request.
So this bounds how long a client waits; no acquisition outlives that wait.