Skip to main content

outer_pull_deadline

Function outer_pull_deadline 

Source
pub fn outer_pull_deadline(per_candidate: Duration, stall: Duration) -> Duration
Expand description

Outer deadline for a node-to-node pull-through, derived from the two configured per-candidate budgets: the stream-open timeout (cache.node_pull_timeout_sec) and the streaming throughput-floor window (cache.node_pull_stall_window_sec).

The delivery handler wraps the whole discover → probe → rank → pull fetch in a single tokio::time::timeout. For the sequential MAX_PROVIDER_ATTEMPTS fallback loop to actually reach candidates #2..N when candidate #1 stalls, this outer deadline must strictly exceed the sum of all per-candidate costs — otherwise both clocks expire together and the outer timeout cancels the whole fetch at the instant candidate #1’s own timeout fires, killing the fallback.

§What one candidate actually costs

THREE sequential bounded stages:

  1. the channel open — CHANNEL_OPEN_CALLER_BUDGET (#1143); then
  2. the stream open — connect → handshake → verified StreamResponse, bounded by per_candidate (#1134); then
  3. streaming, which for a peer that opens honestly and then goes SILENT costs one full throughput-floor window (stall) before the pull gives up on it (#1797).

So the worst case for a candidate is CHANNEL_OPEN_CALLER_BUDGET + per_candidate + stall, and that is what this must budget MAX_PROVIDER_ATTEMPTS of, plus PULL_THROUGH_OUTER_SLACK of one-time discovery overhead.

Every stage has to be a term here. Budgeting only per_candidate would let a cold cache against a slow L2 burn the whole deadline on candidates #1 and #2 and never dial #3. Covering the channel open but not the stall window would leave the same hole for a peer that goes silent mid-stream instead of failing to open — and a worse one, because stall is operator-tunable: at node_pull_stall_window_sec = 120 a single silent candidate outlasts a two-term deadline on its own. Taking stall as an argument keeps this deadline in step with node_pull_stall_window_sec.

§What this deadline does not bound

A candidate’s streaming stage is bounded by inactivity rather than a wall clock, because a wall clock over the bytes caps the blob size a node can pull through (#1134). The stall term above is therefore the cost of a candidate that stops — not a ceiling on one that keeps going. A candidate that is slow but progressing resets that clock on every byte and can consume the whole outer deadline on its own.

That is correct: it is succeeding, and falling through mid-stream would restart the download from zero against another peer, throwing away the bytes already paid for. On expiry the foreground request gives up with a clean miss and nothing continues in the background — there is no detached warm, so a pull only ever runs while a client is waiting, and the blob is re-acquired on the next real client request.

So this bounds how long a client waits; no acquisition outlives that wait.