Module circuit_breaker 

Per-origin half-open circuit-breaker for pull-through (#963).

When an origin is fully unavailable, every cache miss would otherwise run the full retry cycle (up to max_retries with exponential backoff up to max_backoff_ms), serializing that backoff cost across all concurrent misses with no way to fast-fail until the origin recovers. This module adds a per-origin breaker so the cache fails fast and sheds load during a sustained outage, then probes for recovery.

State machine

Standard three-state breaker, one instance per origin in the fallback chain:

• CLOSED — requests flow into the retry loop. Consecutive origin-unavailable failures are counted; on reaching failure_threshold the breaker trips to OPEN. Any success (or a permanent per-object answer like 404) resets the count to zero.
• OPEN — every miss fast-fails before the retry/backoff loop (no backoff incurred) until cooldown_ms has elapsed since the trip, at which point the next acquire transitions to HALF-OPEN.
• HALF-OPEN — up to half_open_max_calls trial pulls are admitted (each running its normal retry budget). A trial success closes the breaker; a trial failure re-opens it and resets the cooldown. Trials beyond the in-flight budget fast-fail like OPEN.

Failure classification

Only origin-unavailable outcomes count toward the threshold — transient failures that exhausted the retry budget. Permanent per-object outcomes (HTTP 404, other 4xx, decode/cap breaches) are recorded as available: the origin responded, it just can’t serve this object. Tripping the breaker on 404s would shed load for content the origin is perfectly able to serve. See OriginOutcome.

Clock injection

Timekeeping goes through the Clock trait so tests drive the cooldown deterministically with ManualClock instead of real sleeps. Production uses SystemClock (monotonic Instant).

Structs

ManualClock

Test clock: a manually-advanced counter. Deterministic — no real time passes. Cloneable so a test can hold one handle and the breaker another, both observing the same advancing value.

OriginBreaker

Per-origin circuit-breaker. One instance fronts each origin’s pull-through retry loop in crate::CacheEngine.

SystemClock

Production clock: monotonic Instant measured from the clock’s construction. Never goes backwards, immune to wall-clock jumps.

TrialGuard

RAII guard for one admitted pull-through attempt.

Enums

Admission

Whether the breaker admits a pull-through attempt right now.

BreakerState

The three breaker states. Exposed for tests and observability; the engine only ever inspects it via OriginBreaker::state.

OriginOutcome

The health of a single per-origin pull-through attempt, as seen by the breaker. The engine maps each origin’s run_with_retry_classified result onto exactly one of these.

Traits

Clock

Monotonic time source for the breaker, injectable so tests can drive the cooldown deterministically rather than sleeping.