Expand description
Origin pull-through retry policy with exponential backoff and jitter (#285).
Wraps a single pull-through attempt in a bounded retry loop. The adapter
(or the engine’s per-attempt closure) classifies each failure as
OriginPullError::Transient or OriginPullError::Permanent; this
module only decides how many transient retries to run and how long
to sleep between them.
§Body-phase retry (#519)
The loop also covers body-phase failures via two paths, both gated on
the same max_retries budget:
- Buffer-then-commit (small blobs): when the adapter advertises a
size_hintat or belowRetryPolicy::buffered_max_bytes(default 4 MiB), the engine’s per-attempt closure drains the stream into a boundedBytesMutbefore handing it to iroh-blobs. Drain errors are classified via the internalclassify_io_errorhelper (which recognises typedBlobTooLargeMarkerandOriginError::*as Permanent, treatsio::ErrorKind::{ConnectionReset, TimedOut, UnexpectedEof, …}and the catch-allOtheras Transient) and re-fed to the loop, so small blobs get mid-stream retries at a bounded memory cost. - Abort + restart (large blobs): the engine’s per-attempt closure
commits to
iroh-blobs::add_streamdirectly. On mid-streamio::Error, the partialTempTagis dropped (iroh-blobs GC reclaims the bytes atcache.gc_interval_seccadence) and the error is classified, restarting the loop from the headers phase. Memory stays bounded; disk amplification is(1 + max_retries) * max_blob_bytesworst case until GC.
§Coalescing interaction and the retry-amplification bound (#1615)
crate::CacheEngine::get coalesces concurrent misses for the same hash
through a single owner task (#305). The retry loop runs inside that owner;
waiters block on the shared Notify until the owner either succeeds or
exhausts. After an owner exhausts, one waiter re-elects itself as the next
owner and runs its own full retry budget — so, taken alone, a burst of N
concurrent waiters on a down origin could drive up to N sequential full
retry budgets.
That amplification is bounded, but by the per-origin circuit-breaker
(crate::circuit_breaker, #963) rather than by anything in this module.
Each budget-exhaustion against a down origin records one Unavailable
outcome; after failure_threshold consecutive exhaustions the breaker
trips OPEN and every later waiter short-circuits before the retry loop
runs. The worst case is therefore failure_threshold full budgets per
origin per cooldown window — a constant, independent of N. The
integration test coalesced_owner_exhaustion_bounded_by_circuit_breaker
pins this, and also pins the O(N) fallback that returns when an operator
disables the breaker.
A tighter per-hash bound — the owner’s failure shared to all waiters so
not even failure_threshold budgets run — is deliberately NOT implemented.
Coalesced waiters do not all have the same origin reach: a populate_local
owner (local origins only, #1116) and a get / populate waiter (full
chain including the paid Peer origin) can coalesce on one hash, and
re-election is load-bearing there — a local-only owner’s NotFound must
not short-circuit a full-chain waiter that could still succeed via Peer.
Sharing the owner’s failure blindly would turn that into a wrong answer;
correct sharing would have to be origin-reach-aware. The circuit-breaker
already sheds the outage load without that complexity, so the shared budget
is not worth its risk on this hot, cancellation-sensitive path.