Skip to main content

Module metrics

Module metrics 

Source
Expand description

Cache-engine OpenMetrics metrics.

Counters and gauges exposed under the decdn_cache_* family per ADR appendix-observability. Two naming rules apply, and they differ by type:

  • Counters are exported with an _total suffix the encoder appends, even though the Rust struct fields here do not carry it — that’s iroh-metrics 1.0.1 encoding.rs:109 writing _total for every Counter. So the field evictions exports as decdn_cache_evictions_total.
  • Gauges (bytes, size_limit_bytes, pinned_count, added with the #1173 eviction driver) get no suffix: the field bytes exports as decdn_cache_bytes.

Never spell _total in a counter field name — it would export as ..._total_total.

The PromQL fragments below use the exported names so operators can paste them verbatim into a Prometheus query.

Operators reason about cache health from four ratios:

  • Hit rate — decdn_cache_hits_total / (decdn_cache_hits_total + decdn_cache_misses_total). The two terms do not share a scope. hits is bumped only inside CacheEngine::get, which the paid serve path never calls — it streams through export_bao_range_stream. misses is also bumped by populate/populate_local and pull_through_range, which the serve-path fill tiers do call. So on a node serving paying clients the numerator is pinned at zero while the denominator climbs with real traffic, and the ratio reads a populated, permanent 0% — not an empty series. Read the node crate’s decdn_serve_cache_{hit,partial_hit,miss}_total for the serve-path hit rate.
  • Origin egress amplification — decdn_cache_pull_through_bytes_total / decdn_bytes_served_total. Approaches — and on a pure pass-through stays just under — 1.0; trends toward 0 as cached content gets re-served. The denominator is the node crate’s wire-write counter rather than decdn_cache_bytes_returned_total, which is get-scoped and holds at zero under a full serve load. The two terms count different byte populations, which is why the pass-through ceiling sits below 1.0 rather than on it: pull_through_bytes is raw origin content, while bytes_served is the bao wire form — content plus interleaved proof nodes (≈0.4%), over a range widened to enclosing 16 KiB chunk-group boundaries. Read the ratio as a trend, not against an exact 1.0.
  • Origin retry health — decdn_cache_origin_retry_exhausted_total / decdn_cache_origin_fetches_total. Sustained nonzero rate = user-visible origin failures the retry budget couldn’t save (#285).
  • GC reclaim ratio — rate(decdn_cache_gc_bytes_reclaimed_total) / rate(decdn_cache_pull_through_bytes_total) over a recent observation window. pull_through_bytes counts every origin byte we receive, success or failure (#418), so this ratio approaches 1.0 in the hostile-origin amplification scenario where almost everything we pull is later reclaimed by GC. Healthy nodes sit near 0.0. Use a window at least 2 * cache.gc_interval_sec wide: byte attribution lags one sweep cycle (see gc_bytes_reclaimed).

Hit/miss accounting (#418): on Ok and on the cache-domain error returns (NoOrigin, evicted NotFound, origin NotFound, HashMismatch, BlobTooLarge, OriginError), CacheEngine::get bumps exactly one of hits or misses.

Store I/O error carve-out (CacheError::Store): these surface via tracing::error! and split by phase.

  • Local-lookup phase (has, read_local before pull-through is initiated): a Store error here means the cache can’t determine whether the blob is present, so neither counter is bumped. A degraded local store does not silently classify reads as misses.
  • Pull-through phase (after the miss is already determined — e.g., add_bytes fails to persist verified origin bytes): the misses and pull_through_bytes bumps that fired earlier in pull_through stay bumped. The miss genuinely happened, the origin egress was genuinely paid, and the failure surfaces both via tracing::error! and as the get-call’s Err.

The cache crate owns this group rather than re-exporting node-side state so the cache stays self-describing. Node wires Arc<CacheMetrics> into its registry under the decdn_cache prefix.

Structs§

CacheMetrics
OpenMetrics counters and gauges for the cache engine. Register via Registry::register(Arc::clone(&cache_metrics) as Arc<dyn MetricsGroup>) (or under a sub-registry prefix); the same Arc is handed to crate::CacheEngine::open_full so engine-side bumps land in the same encoder output.