Struct CacheMetrics 

pub struct CacheMetrics {

    pub origin_fetches: Counter,
    pub fill_not_coalesced: Counter,
    pub origin_retry_exhausted: Counter,
    pub origin_fallback: Counter,
    pub hits: Counter,
    pub misses: Counter,
    pub bytes_returned: Counter,
    pub pull_through_bytes: Counter,
    pub gc_runs: Counter,
    pub gc_bytes_reclaimed: Counter,
    pub tag_drop_failures: Counter,
    pub origin_probe_failures: Counter,
    pub origin_enumerate_failures: Counter,
    pub circuit_breaker_trips: Counter,
    pub circuit_breaker_recoveries: Counter,
    pub circuit_breaker_short_circuits: Counter,
    pub bytes: Gauge,
    pub size_limit_bytes: Gauge,
    pub pinned_count: Gauge,
    pub evictions: Counter,
    pub evictions_bytes: Counter,
    pub evictions_starved: Counter,
    pub size_measure_failures: Counter,
    pub recency_seed_failures: Counter,
    pub partial_size_observe_failures: Counter,
    pub inflight_mutex_poisoned: Counter,
    pub evicted_operator: Counter,
    pub held_corruption_quarantined: Counter,
    pub origin_range_timeouts: Counter,
    pub range_pull_permit_waits: Counter,
}

OpenMetrics counters and gauges for the cache engine. Register via Registry::register(Arc::clone(&cache_metrics) as Arc<dyn MetricsGroup>) (or under a sub-registry prefix); the same Arc is handed to crate::CacheEngine::open_full so engine-side bumps land in the same encoder output.

Fields

origin_fetches: Counter

Origin pull-through fetches attempted (one per pull_through call, success or failure). Denominator for any retry-exhaustion alert.

fill_not_coalesced: Counter

Serve-miss fill claims NOT coalesced onto a live same-hash fill whose covered range overlaps the request, because the request starts ahead of that fill’s paid frontier (#2062): attaching would park the request on a pull that advances only as the other client pays. Counted once per claim. The refused attach opens its own pull, and the part of the overlap not yet in the store when that pull starts is fetched twice — duplicate egress (own-origin S3, or upstream USDC on the peer tier) in exchange for liveness. An overlap already in the store is never fetched again. Emitted name: decdn_cache_fill_not_coalesced_total. Operator-actionable: a sustained rate means clients habitually resume ahead of what payers have cleared.

origin_retry_exhausted: Counter

Origin fetches that gave up after exhausting max_retries (#285). Operator-actionable: any nonzero rate = user-visible origin failures the retry budget couldn’t save.

origin_fallback: Counter

Times pull_through advanced from one origin to the next in the operator-configured fallback chain (#284). One bump per chain-walk step, not per per-origin retry — the latter is covered by origin_retry_exhausted. Bumped only when there is a next entry to try, so a single-origin chain (the common case) keeps this counter flat at zero. The denominator for a “what fraction of misses needed fallback” alert is decdn_cache_origin_fetches_total. Sustained nonzero against a healthy primary backend suggests the chain is masking an outage that should be paged on instead.

hits: Counter

get() calls served from the local store (#418). Includes both first-attempt hits and waiter retries that find the blob present after a coalesced concurrent pull-through completes (the engine’s coalescing is between in-process tasks fronting the same hash, not between network peers). Paired with misses; see the module-level doc for the hits-XOR-misses contract and the store-error carve-out.

misses: Counter

get() calls that did not find the blob locally (#418). Bumped once on every operator-evicted NotFound return and once at the entry of every pull-through — the latter covers all pull-through outcomes (Ok and any later error, including a Store error from add_bytes) without enumerating each variant. See the module-level doc for the contract with hits and the per-phase store-error carve-out.

bytes_returned: Counter

Total bytes returned to the caller of get() on success (#418). Counts both cache-hit and pull-through-success paths.

pull_through_bytes: Counter

Bytes fetched from origin during pull-through (#418). Counted the moment bytes are received from origin, regardless of whether they pass BLAKE3 verification or land in the store — origin egress is paid either way. On the range-pull path this includes each {H}.obao4 outboard the origin serves, wrong-length copies too; a cached outboard is not re-counted. Distinguishes ‘serving from cache’ vs. ‘paying origin egress’ when paired with bytes_returned.

gc_runs: Counter

iroh-blobs GC sweep cycles observed (#518). Bumped once per add_protected callback fire, after the pre-sweep snapshot succeeds. Iroh-blobs spawns the sweep loop internally when cache.gc_interval_sec > 0; the cb is our hook into each cycle. On-demand reclamation is tracked under #520.

Operator-actionable: a flat-line at zero against a nonzero cache.gc_interval_sec means the periodic loop never spawned, or the snapshot has been failing on every cycle (the engine emits a tracing::warn! when that happens).

Field name omits _total: the OpenMetrics encoder appends it automatically (iroh-metrics 1.0.1 encoding.rs:109), so the emitted name is decdn_cache_gc_runs_total.

gc_bytes_reclaimed: Counter

Bytes reclaimed by the iroh-blobs GC, attributed across cycles (#518). Computed as the pre-sweep blob-set diff between the previous cycle and the current cycle: hashes that vanish across that window are exactly what the previous sweep deleted (the cache crate has no other public delete path). Sustained nonzero rate against zero rate(pull_through_bytes) is a red flag — either the pull path is failing to promote named tags on success, or a hostile origin is amplifying disk usage via repeated mid-stream errors.

Attribution lags one sweep cycle. The first cycle records a baseline and bumps zero bytes; from cycle two onward the bump reflects the previous sweep’s reclaim. For an alert window shorter than 2 * cache.gc_interval_sec the reading will be noisy.

Field name omits _total: the OpenMetrics encoder appends it automatically, so the emitted name is decdn_cache_gc_bytes_reclaimed_total.

tag_drop_failures: Counter

Best-effort named-tag deletions that failed (#860/#837). Bumped when evict() (DMCA/corruption takedown) or the drain-path hash-mismatch arm cannot delete a blob’s protecting named tag. Serving is unaffected — an evict’s logical-evicted set still blocks get/has, and a mismatch still returns HashMismatch — but the bytes stay GC-protected on disk because the tag survives, and nothing re-attempts the delete.

Operator-actionable: any sustained nonzero rate means disk reclaim is stuck. For an evict() failure that is a DMCA/compliance concern (the takedown stopped serving but did not reclaim the bytes); for the drain path it is the #837 unbounded-disk-growth leak under a hostile origin. A persistently nonzero rate warrants investigating the iroh-blobs tag store (a wedged store actor, I/O errors); the engine also emits a tracing::warn! per failure.

Field name omits _total: the OpenMetrics encoder appends it automatically, so the emitted name is decdn_cache_tag_drop_failures_total.

origin_probe_failures: Counter

Origin size probes that faulted during an origin rescan — a transport error from one or more configured origins with none of them confirming the object, or the probe walk overrunning its ceiling — rather than answering “held” or “not held”.

Operator-actionable: a rescan resolves each candidate against the origin to decide what this node advertises, so a throttled HeadObject window during a rescan would otherwise shrink the announce set silently. A retry-eligible fault keeps whatever entry an earlier pass indexed, so a sustained nonzero rate means the node may still be advertising content the origin has since stopped holding — every request for it is then a refusal. A candidate first seen inside the fault window has no earlier entry and is simply absent until a later rescan resolves it. Correlate with the origin backend’s own error rate.

Field name omits _total: the OpenMetrics encoder appends it automatically, so the emitted name is decdn_cache_origin_probe_failures_total.

origin_enumerate_failures: Counter

Origins whose enumerate failed during a rescan, counted once per origin per rescan.

The sibling of origin_probe_failures for the other leg of the same rescan, and the more severe of the two: a failed listing contributes no candidates at all, so every hash discoverable only through that origin leaves the announce set with no per-hash fault to carry a previous entry forward. Only the operator pins naming those hashes survive. Operator- actionable in the same way — correlate with the origin backend’s error rate; a later successful rescan restores the listing.

Field name omits _total: the OpenMetrics encoder appends it automatically, so the emitted name is decdn_cache_origin_enumerate_failures_total.

circuit_breaker_trips: Counter

Per-origin circuit-breaker trips from CLOSED/HALF-OPEN to OPEN (#963). Bumped once each time a breaker opens — on crossing the failure_threshold from CLOSED, or on a failed HALF-OPEN trial. Sustained nonzero rate against a configured origin means that origin is having a sustained outage and the cache is shedding the retry-backoff storm by fast-failing its misses.

Operator-actionable: a breaker that keeps opening points at an unhealthy origin (or a backend the chain is masking). Pair with circuit_breaker_short_circuits to see how much retry-backoff work the breaker saved while open.

Field name omits _total: the OpenMetrics encoder appends it automatically, so the emitted name is decdn_cache_circuit_breaker_trips_total.

circuit_breaker_recoveries: Counter

Per-origin circuit-breaker recoveries from HALF-OPEN to CLOSED (#963). Bumped once each time a half-open trial succeeds and the breaker closes. A trip followed by a recovery is the healthy outage→recovery cycle; trips without matching recoveries means an origin that keeps failing its half-open probes.

Field name omits _total: the emitted name is decdn_cache_circuit_breaker_recoveries_total.

circuit_breaker_short_circuits: Counter

Cache misses fast-failed by an OPEN circuit-breaker before the retry/backoff loop ran (#963). Each bump is one miss that would otherwise have incurred up to the full max_retries-worth of exponential backoff against a dead origin; the breaker shed that cost. The whole point of the feature is to drive this counter up during an outage so the retry-exhaustion and origin-egress counters stay flat.

Field name omits _total: the emitted name is decdn_cache_circuit_breaker_short_circuits_total.

bytes: Gauge

Current on-disk cache footprint in bytes (decdn_cache_bytes). Set by the eviction driver each sweep from crate::CacheEngine::total_bytes. Pairs with size_limit_bytes for a saturation ratio; when it approaches size_limit_bytes the cache is at its high-water mark and the driver is actively evicting.

size_limit_bytes: Gauge

Effective cache ceiling in bytes (decdn_cache_size_limit_bytes) — the denominator of the saturation ratio against bytes. The eviction driver sets it every tick: cache.cache_size_mb × 1 048 576, clamped down so the cache never grows into the last cache.disk_headroom_mb of free disk on the volume (#1930). It therefore moves with real free space, and reads below the configured budget whenever the disk clamp is the binding ceiling.

pinned_count: Gauge

Size of the operator-pinned set (decdn_cache_pinned_count). Set by the eviction driver each sweep from crate::CacheEngine::pinned_snapshot. Pinned hashes are LRU-exempt, so a pinned set approaching the cache size is a starvation risk.

evictions: Counter

LRU eviction victims actually released under cache-size pressure (decdn_cache_evictions_total). Bumped once per hash for which crate::CacheEngine::release_for_eviction reported a real release (Ok(n > 0)); a pinned refusal or an untagged blob returns Ok(0) and is deliberately not counted, so this measures releases rather than attempts. LRU pressure only — operator takedowns are counted separately by evicted_operator.

A released blob keeps serving until the iroh-blobs GC reclaims it, and a serve re-inserts its access-time entry, so the same hash can legitimately be re-selected and re-released across ticks.

Field name omits _total: the OpenMetrics encoder appends it, so the emitted name is decdn_cache_evictions_total.

evictions_bytes: Counter

Bytes released by the eviction driver (decdn_cache_evictions_bytes_total), summed from the per-hash sizes of successful releases. These are bytes made GC-eligible, not yet reclaimed — pair with bytes to see reclaim actually landing.

Field name omits _total: the emitted name is decdn_cache_evictions_bytes_total.

evictions_starved: Counter

Eviction passes that made no progress while still over eviction_target_pct (decdn_cache_evictions_starved_total). Two causes, both operator-actionable but with different remedies:

• crate::CacheEngine::eviction_candidates returned empty. That set is filtered on pinned + probe-held, so the remedy is to raise cache.cache_size_mb, lower max_probe_holds, or trim the pinned set. The set is also short when the open-time store walk failed (Self::recency_seed_failures): blobs from before the restart are not candidates until accessed. There the remedy is to restart once the store is healthy.
• The pass had candidates but released none (all pinned, untagged, or erroring).

A sustained nonzero rate with bytes ≈ size_limit_bytes is the alarm.

Field name omits _total: the emitted name is decdn_cache_evictions_starved_total.

size_measure_failures: Counter

Cache-size measurement failures in the eviction driver (decdn_cache_size_measure_failures_total) — a failed total_bytes() or size_snapshot() store walk. Operator-actionable: while nonzero, the bytes gauge is stale (the driver returns before refreshing it) and the ceiling is not being enforced, so a saturation dashboard reading off bytes alone will look healthy while the cache grows.

Field name omits _total: the emitted name is decdn_cache_size_measure_failures_total.

recency_seed_failures: Counter

Failed store walks while seeding eviction recency at cache open (decdn_cache_recency_seed_failures_total). Open still succeeds, but until the next restart blobs from before the restart are not eviction candidates until accessed. Over the size limit the driver then releases only blobs accessed since open, or starves (evictions_starved) if there are none. Nonzero is the alarm; restart once the store is healthy.

Field name omits _total: the emitted name is decdn_cache_recency_seed_failures_total.

partial_size_observe_failures: Counter

Partial blobs whose observe() failed during a cache-size walk (decdn_cache_partial_size_observe_failures_total). The walk counts such a blob at its last known present bytes, else at status()’s size, and keeps going, so bytes stays fresh but can be off for that blob. The walk retries a failed blob once per partial-size TTL, so a store entry that stays broken ticks this steadily. The warn! beside each tick names the hash.

Field name omits _total: the emitted name is decdn_cache_partial_size_observe_failures_total.

inflight_mutex_poisoned: Counter

Times the in-flight fill-coalescing mutex was found poisoned (#1517). A poison means some earlier task panicked while holding the map guard; the workspace anti-panic policy (unwrap_used / expect_used / panic denied) makes that close to unreachable, so any nonzero value is a genuine bug worth filing rather than an operational condition worth tuning.

The engine recovers the guard via PoisonError::into_inner, clears the poison, and keeps coalescing — so this is a report, not a degradation: it does not mean duplicate origin pulls happened, and it does not call for a restart. It is metered because the alternative — discarding the error — silently disables coalescing for the life of the process, and the only observable is decdn_cache_pull_through_bytes_total rising faster than request volume.

Counts poisonings, not locks-since-a-poisoning. Because the engine clears the poison, one panic bumps this exactly once; a rate() panel therefore shows the incidents rather than the request rate that followed them. Paired with a single tracing::error!, latched so a pathological panic loop cannot spam the log while this counter keeps the true total.

Note: the bump is skipped when the engine was built with no metrics handle (CacheEngine::open’s 3-arg form, test-only today — the daemon always wires Some(..)). There the latched log line is the whole signal.

Field name omits _total: the OpenMetrics encoder appends it automatically, so the emitted name is decdn_cache_inflight_mutex_poisoned_total.

evicted_operator: Counter

Hashes removed via the durable operator-evict path (decdn_cache_evicted_operator_total) — decdn node evict / DMCA takedown. Bumped once per takedown that actually stopped serving: an idempotent repeat-evict short-circuits on the already-evicted check and returns Ok(()) without bumping, so this counts distinct hashes, not calls. Distinct from evictions (LRU pressure): this is a permanent, evicted.log-backed removal, not cache-size management.

Field name omits _total: the emitted name is decdn_cache_evicted_operator_total.

held_corruption_quarantined: Counter

Held hashes quarantined because a serve export failed bao validation against the content root: the stored bytes or outboard diverged after admission. Bumped once per quarantine, not once per failed serve. Distinct from evicted_operator: a quarantine is not durable, it releases the entry to GC, and the hash is re-acquirable once the sweep reclaims it.

Operator-actionable: a nonzero rate means disk rot or tampering under the cache directory. Check the disk, and confirm that GC is enabled so the corrupt entries are reclaimed.

Field name omits _total: the emitted name is decdn_cache_held_corruption_quarantined_total.

origin_range_timeouts: Counter

Origin reads on the range-pull path (an {H}.obao4 fetch or one data window) that ran past their time budget (ADR 037). The budget is cache.node_pull_stall_window_sec plus the read size at cache.node_pull_min_throughput_bps. Each timeout is also an origin transport fault, so it advances the origin chain or fails the fill.

Operator-actionable: a nonzero rate means an origin that stalls or sustains less than the throughput floor. Check the origin, or lower the floor for a slow origin.

Field name omits _total: the emitted name is decdn_cache_origin_range_timeouts_total.

range_pull_permit_waits: Counter

Own-origin range draws that found the engine-wide pool of MAX_CONCURRENT_RANGE_PULLS permits full and waited for one. Every draw of every fill shares the pool, and a hung origin holds a permit for a whole draw, so a starved fill shows here and not only as a client stall. A wait past RANGE_PULL_PERMIT_WARN_AFTER also logs a warning.

Operator-actionable: a sustained rate means more concurrent own-origin fills than the pool serves, or an origin slow enough to hold permits. Check origin_range_timeouts and the origin’s latency.

Field name omits _total: the emitted name is decdn_cache_range_pull_permit_waits_total.

Trait Implementations

impl Debug for CacheMetrics

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for CacheMetrics

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for CacheMetrics

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Iterable for CacheMetrics

fn metric_field_count(&self) -> usize

Returns the number of metric fields in the struct.

fn metric_field_ref(&self, n: usize) -> Option<MetricItem<'_>>

Returns the metric field at the given index.

fn family_field_count(&self) -> usize

Returns the number of Family fields in the struct.

fn family_field_ref(&self, _n: usize) -> Option<FamilyItem<'_>>

Returns the Family field at the given index.

impl MetricsGroup for CacheMetrics

fn name(&self) -> &'static str

Returns the name of this metrics group.

fn iter(&self) -> FieldIter<'_>

Returns an iterator over all metric items with their values and helps.

impl Serialize for CacheMetrics

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.