Expand description
Cache-engine OpenMetrics metrics.
Counters and gauges exposed under the decdn_cache_* family per ADR
appendix-observability. Two naming rules apply, and they differ by type:
- Counters are exported with an
_totalsuffix the encoder appends, even though the Rust struct fields here do not carry it — that’siroh-metrics 1.0.1 encoding.rs:109writing_totalfor everyCounter. So the fieldevictionsexports asdecdn_cache_evictions_total. - Gauges (
bytes,size_limit_bytes,pinned_count, added with the #1173 eviction driver) get no suffix: the fieldbytesexports asdecdn_cache_bytes.
Never spell _total in a counter field name — it would export as
..._total_total.
The PromQL fragments below use the exported names so operators can paste
them verbatim into a Prometheus query.
Operators reason about cache health from four ratios:
- Hit rate —
decdn_cache_hits_total / (decdn_cache_hits_total + decdn_cache_misses_total). The two terms do not share a scope.hitsis bumped only insideCacheEngine::get, which the paid serve path never calls — it streams throughexport_bao_range_stream.missesis also bumped bypopulate/populate_localandpull_through_range, which the serve-path fill tiers do call. So on a node serving paying clients the numerator is pinned at zero while the denominator climbs with real traffic, and the ratio reads a populated, permanent 0% — not an empty series. Read the node crate’sdecdn_serve_cache_{hit,partial_hit,miss}_totalfor the serve-path hit rate. - Origin egress amplification —
decdn_cache_pull_through_bytes_total / decdn_bytes_served_total. Approaches — and on a pure pass-through stays just under — 1.0; trends toward 0 as cached content gets re-served. The denominator is the node crate’s wire-write counter rather thandecdn_cache_bytes_returned_total, which isget-scoped and holds at zero under a full serve load. The two terms count different byte populations, which is why the pass-through ceiling sits below 1.0 rather than on it:pull_through_bytesis raw origin content, whilebytes_servedis the bao wire form — content plus interleaved proof nodes (≈0.4%), over a range widened to enclosing 16 KiB chunk-group boundaries. Read the ratio as a trend, not against an exact 1.0. - Origin retry health —
decdn_cache_origin_retry_exhausted_total / decdn_cache_origin_fetches_total. Sustained nonzero rate = user-visible origin failures the retry budget couldn’t save (#285). - GC reclaim ratio —
rate(decdn_cache_gc_bytes_reclaimed_total) / rate(decdn_cache_pull_through_bytes_total)over a recent observation window.pull_through_bytescounts every origin byte we receive, success or failure (#418), so this ratio approaches1.0in the hostile-origin amplification scenario where almost everything we pull is later reclaimed by GC. Healthy nodes sit near0.0. Use a window at least2 * cache.gc_interval_secwide: byte attribution lags one sweep cycle (seegc_bytes_reclaimed).
Hit/miss accounting (#418): on Ok and on the cache-domain error
returns (NoOrigin, evicted NotFound, origin NotFound,
HashMismatch, BlobTooLarge, OriginError), CacheEngine::get
bumps exactly one of hits or misses.
Store I/O error carve-out (CacheError::Store): these surface via
tracing::error! and split by phase.
- Local-lookup phase (
has,read_localbefore pull-through is initiated): a Store error here means the cache can’t determine whether the blob is present, so neither counter is bumped. A degraded local store does not silently classify reads as misses. - Pull-through phase (after the miss is already determined —
e.g.,
add_bytesfails to persist verified origin bytes): themissesandpull_through_bytesbumps that fired earlier inpull_throughstay bumped. The miss genuinely happened, the origin egress was genuinely paid, and the failure surfaces both viatracing::error!and as the get-call’sErr.
The cache crate owns this group rather than re-exporting node-side
state so the cache stays self-describing. Node wires Arc<CacheMetrics>
into its registry under the decdn_cache prefix.
Structs§
- Cache
Metrics OpenMetricscounters and gauges for the cache engine. Register viaRegistry::register(Arc::clone(&cache_metrics) as Arc<dyn MetricsGroup>)(or under a sub-registry prefix); the sameArcis handed tocrate::CacheEngine::open_fullso engine-side bumps land in the same encoder output.