Skip to main content

Module retry

Module retry 

Source
Expand description

Origin pull-through retry policy with exponential backoff and jitter (#285).

Wraps a single pull-through attempt in a bounded retry loop. The adapter (or the engine’s per-attempt closure) classifies each failure as OriginPullError::Transient or OriginPullError::Permanent; this module only decides how many transient retries to run and how long to sleep between them.

§Body-phase retry (#519)

The loop also covers body-phase failures via two paths, both gated on the same max_retries budget:

  • Buffer-then-commit (small blobs): when the adapter advertises a size_hint at or below RetryPolicy::buffered_max_bytes (default 4 MiB), the engine’s per-attempt closure drains the stream into a bounded BytesMut before handing it to iroh-blobs. Drain errors are classified via the internal classify_io_error helper (which recognises typed BlobTooLargeMarker and OriginError::* as Permanent, treats io::ErrorKind::{ConnectionReset, TimedOut, UnexpectedEof, …} and the catch-all Other as Transient) and re-fed to the loop, so small blobs get mid-stream retries at a bounded memory cost.
  • Abort + restart (large blobs): the engine’s per-attempt closure commits to iroh-blobs::add_stream directly. On mid-stream io::Error, the partial TempTag is dropped (iroh-blobs GC reclaims the bytes at cache.gc_interval_sec cadence) and the error is classified, restarting the loop from the headers phase. Memory stays bounded; disk amplification is (1 + max_retries) * max_blob_bytes worst case until GC.

§Coalescing interaction and the retry-amplification bound (#1615)

crate::CacheEngine::get coalesces concurrent misses for the same hash through a single owner task (#305). The retry loop runs inside that owner; waiters block on the shared Notify until the owner either succeeds or exhausts. After an owner exhausts, one waiter re-elects itself as the next owner and runs its own full retry budget — so, taken alone, a burst of N concurrent waiters on a down origin could drive up to N sequential full retry budgets.

That amplification is bounded, but by the per-origin circuit-breaker (crate::circuit_breaker, #963) rather than by anything in this module. Each budget-exhaustion against a down origin records one Unavailable outcome; after failure_threshold consecutive exhaustions the breaker trips OPEN and every later waiter short-circuits before the retry loop runs. The worst case is therefore failure_threshold full budgets per origin per cooldown window — a constant, independent of N. The integration test coalesced_owner_exhaustion_bounded_by_circuit_breaker pins this, and also pins the O(N) fallback that returns when an operator disables the breaker.

A tighter per-hash bound — the owner’s failure shared to all waiters so not even failure_threshold budgets run — is deliberately NOT implemented. Coalesced waiters do not all have the same origin reach: a populate_local owner (local origins only, #1116) and a get / populate waiter (full chain including the paid Peer origin) can coalesce on one hash, and re-election is load-bearing there — a local-only owner’s NotFound must not short-circuit a full-chain waiter that could still succeed via Peer. Sharing the owner’s failure blindly would turn that into a wrong answer; correct sharing would have to be origin-reach-aware. The circuit-breaker already sheds the outage load without that complexity, so the shared budget is not worth its risk on this hot, cancellation-sensitive path.