Skip to main content

Metrics

Struct Metrics 

Source
pub struct Metrics { /* private fields */ }
Expand description

Aggregated deCDN node metrics.

Implementations§

Source§

impl Metrics

Source

pub fn new() -> Self

Create the registry and register deCDN’s metric group plus the cache crate’s decdn_cache_* group. The cache handle is shared with the engine via Self::cache_metrics so engine-side bumps land in the same encoder output.

Source

pub fn cache_metrics(&self) -> Arc<CacheMetrics> ⓘ

Shared Arc<CacheMetrics> for wiring into decdn_cache::CacheEngine.

Source

pub fn register_iroh_endpoint(&self, ep: &Endpoint) -> Result<()>

Register iroh’s transport metrics under the decdn_iroh_ prefix so socket_*, net_report_*, etc. come out as decdn_iroh_socket_*, matching adr/appendix-observability.md’s naming convention.

§Errors

Returns an error if the registry lock is poisoned.

Source

pub fn pool_open_failure_by_reason(&self, reason: PoolOpenFailureReason)

Record a buyer openChannel-tx failure broken out by cause (#966): bumps the decdn_pool_open_failures_{reason}_total sibling counter for reason. Pairs with the structured reason field on the warn!/debug! in crate::node_origin. Distinct from Self::node_pull_pool_open_failure, the unlabeled total (which also counts store/expired-reclaim causes that never reach the openChannel tx).

Source

pub fn serve_economics_refused(&self, regime: ServeEconomicsRegime)

Record an ADR 041 serve-economics refusal: bumps the decdn_serve_economics_refused_total aggregate and the matching regime sibling. Enum-dispatch shape of Self::pool_open_failure_by_reason; both counters are plain siblings on self.decdn, so they export at zero from a fresh registry without pre-materialization. Pairs with the structured debug! at the refusal site in crate::node_origin.

Source

pub fn load_shed_refused(&self, reason: ShedReason)

Record a load-shed refusal broken out by cause into its sibling counter (decdn_load_shed_refused_*_total). Per adr/appendix-observability.md § Reason splits each shed reason is a sibling, not a label, because the remedies differ. The exhaustive match is the gate: a new crate::load_shed::ShedReason variant fails to compile until it is given a counter here. Pairs with the structured debug! at each shed site in crate::handlers::client.

Source

pub fn probe_hold_unavailable(&self, reason: ProbeHoldUnavailableReason)

Record a probe that could not be answered from a guaranteed eviction hold, broken out by cause on the reason label of decdn_probe_hold_unavailable_total (#1443). Hand-written rather than a recorders! entry because the pre-materialized child handles live on Metrics, whereas recorders! only reaches self.decdn.$field. Same enum-dispatch shape as Self::pool_open_failure_by_reason, though that one’s counters are siblings on self.decdn. Pairs with the structured debug! at each call site in crate::handlers::probe.

Source

pub fn staker_set_active_by_region( &self, counts: &BTreeMap<Region, usize>, unknown: usize, )

Publish the active-staker set size per region: one decdn_staker_set_active_by_region child per entry in counts, plus unknown on decdn_staker_set_active_unknown_region. A region absent from counts has its child removed, so the map shows no stale country. Stale children are removed after the live ones are set, so a concurrent scrape never sees the family emptied part-way through an update. It can still see some regions at their new value and others at the old one.

Source

pub fn dispatch_in_flight_value(&self) -> u64

Current value of the dispatch_in_flight gauge as a u64. Read by admin_v1_health.in_flight_streams (issue #604) so the decdn node drain --wait client can observe in-flight client streams reach 0 during a graceful drain.

A negative gauge reading is forbidden by the Permit acquire/release RAII pair: dispatch_permit_acquired runs once before each Permit exists and dispatch_permit_released runs once on Drop. If the conversion fails the pair has been broken — we both debug_assert! (loud failure in tests/dev), log tracing::error! (visible in production), and return 0 rather than wrapping to a huge u64. Returning 0 here would be read by decdn node drain --wait as “drain complete” while real streams are still in flight, so the loud log is the operator-actionable signal that something broke upstream.

Source

pub fn connection_guard(&self) -> ConnectionGuard<'_>

RAII guard that increments active_connections on construction and decrements it on drop, so the gauge stays correct even if the handler future is cancelled between open and close.

Source

pub fn encode(&self) -> Result<String>

Render the registry as OpenMetrics text — exactly the body the public /metrics HTTP endpoint serves. pub (not pub(crate)) so integration tests in sibling crates can assert on the exported series without scraping over TCP; it exposes no data the unauthenticated /metrics endpoint doesn’t already.

Source§

impl Metrics

Source

pub fn probe_request(&self)

A cdn/probe/v1 request was served.

Source

pub fn probe_hold_slots_max(&self, max: usize)

Publish the configured max_probe_holds budget (registry-mandatory decdn_probe_hold_slots_max). Called once at runtime bring-up.

Source

pub fn voucher_nonce_gap(&self)

An accepted voucher skipped one or more nonce values past last_nonce + 1 (#747). Counted once per gapped voucher; the precise skip count rides the paired tracing::warn! in apply_voucher.

Source

pub fn redeem_hint_dropped(&self)

A redeem hint was dropped because the bounded advisory channel was full (try_send → Full, #751). Advisory, so a few drops are benign; a sustained rate means the redeemer is not keeping up with fan-out.

Source

pub fn redeem_hint_parked(&self)

A redeem hint was dropped because its lane is parked after a chain fault on the hint path (#2340). The next sweep retries the lane.

Source

pub fn receipt_write_dropped(&self)

A download-receipt audit write was dropped because the bounded writer queue was full (try_send → Full, #803). Audit-only, so a drop never affects settlement; a sustained rate means the writer is not keeping up with disk I/O and audit records are being lost.

Source

pub fn warming_credit_dropped(&self)

An ADR 041 warming serve credit was dropped before it reached the ledger — the bounded aggregator queue was full, or the aggregator was gone. The drop is conservative (the source ledger stays lower than reality), but a sustained rate throttles speculative warming.

Source

pub fn warming_credit_applied(&self)

An ADR 041 warming serve credit reached the ledger. Paired with warming_credit_dropped so an operator can tell “no credits dropped” from “no credits at all”.

Source

pub fn redemption_failure(&self)

A redemption step failed: a redeemMany chunk reverted or was refused at send, or a lane could not be planned or loaded. Pairs with the warn! at each site.

Source

pub fn redemption_skipped_insolvent(&self)

A lane was held or dropped by pool_is_redeemable because its pool’s chain-observed solvency ruled it out this pass (ADR 003).

Source

pub fn redemption_skipped_expired(&self)

A lane with value owed was skipped this pass because its signer’s capability has expired, or expires within the redeemer’s landing slack (ADR 003 §Revocation).

Source

pub fn redemption_skipped_insolvent_by(&self, n: u64)

n lanes were held or dropped by pool_is_redeemable in one planning pass (ADR 003); the batched form of redemption_skipped_insolvent.

Source

pub fn redemption_reconciled_skip_by(&self, n: u64)

n lanes were dropped from a redeem batch by the pre-submit on-chain watermark reconciliation because the chain already shows them settled to their claim value (the redeemMany no-op guard, avoided before it costs gas).

Source

pub fn redemption_reconcile_ok(&self)

A pre-submit on-chain watermark batch landed and was reconciled against the chain — the attempt signal that makes a zero redemption_reconciled_skip readable.

Source

pub fn redemption_reconcile_failure(&self)

A pre-submit on-chain watermark batch did not land (RPC error, timeout, or a mismatched return length). Fail-open: the lanes it could not read are submitted unchanged.

Source

pub fn buyer_reclaim_failure(&self)

A buyer-side reclaim-sweep attempt (try_reclaim) failed — an RPC/receipt error, an on-chain revert, or a failed store write when clearing the local record (#906). Pairs with the per-attempt warn! in try_reclaim and the threshold error! in reclaim_once.

Source

pub fn buyer_pool_store_skipped_undecodable_records(&self, count: usize)

Buyer-pool rows skipped as undecodable during one successful store hydration (#1271). Counted once per skipped row per load attempt (each of the startup, reconcile, and reclaim loads bumps it), so a persistent malformed row keeps the escrowed-but-untracked alert active. The usize count is saturated into the u64 counter.

Source

pub fn buyer_topup_ok(&self)

A background low-water top-up (#1146) landed: a reused buyer channel below its 20% low-water mark was re-funded to the working deposit. Pairs with the info! in spawn_refill_if_low.

Source

pub fn buyer_topup_failure(&self)

A background low-water top-up (#1146) did not cleanly land — the allowance re-approval or the topUp submit/receipt errored or reverted, OR the topUp landed on-chain but the local row vanished/rotated during the RPC (DepositOutcome::UnknownProvider / ChannelMismatch, i.e. escrowed-but- untracked). Folding the untracked case in here keeps stranded deposits visible on this counter. Best-effort, so the channel is left un-topped; pairs with the warn! (or top_up’s own error!/warn!) around the call in spawn_refill_if_low.

Source

pub fn watcher_persist_failure(&self)

The settlement watcher failed to forget a reclaimed pool’s lane state (forget_lane) on a PoolReclaimed event, in PoolSettlementSink (payment_settlement.rs). The failure is swallowed — logged with a per-site warn! (“failed to forget reclaimed lane”) and the cursor advances — so it is a lane-store health signal, not a stuck settlement.

Source

pub fn lane_flush_failure(&self)

A lane-store background flush failed — the dirty in-memory set is retained and retried on the next timer tick, or a final flush on shutdown failed. Pairs with the warn!s in the background flush task and the shutdown flush in runtime/mod.rs.

Source

pub fn slash_detected(&self)

A distinct slash against this node’s operator was detected by the slash watcher (#1032). Counts each slashId once (backfill + live dedup).

Source

pub fn slash_resync_failure(&self)

A slash-watcher resync failed and the current detected-slash set was kept. Bumps slash_resync_failures_total.

Source

pub fn staker_set_watcher_resolve_failure(&self)

A nodeIdOf(operator) resolution for an operator-indexed event failed, dropping the membership change (#788, crate::dht::chain_staker_set). Bumps staker_set_watcher_resolve_failures_total. Pairs with the per-failure warn! in crate::dht::capacity_bond_registry’s RegistrySink::on_operator_change.

Source

pub fn staker_set_active_count(&self, count: usize)

Publish the current cached active-staker set size (#783). Sampled on every membership change the watcher applies, so the gauge tracks the cached view — which under a watcher outage is exactly the (possibly stale) set that admission decisions read.

Source

pub fn node_address_directory_size(&self, count: usize)

Publish the current cached NodeId → operator address binding count (#831). Sampled on every binding change the node-address watcher applies, so it tracks the cached view the pull path resolves against.

Source

pub fn node_pull_attempt(&self)

A node-to-node pull orchestration found ≥1 candidate and is attempting a fill (#831).

Source

pub fn node_pull_success(&self)

A node-to-node pull delivered verified bytes (#831).

Source

pub fn node_pull_no_providers(&self)

A cache miss surfaced no provider from discovery (#831).

Source

pub fn probe_cache_hit(&self)

A cache-miss pull was served from the ADR 001 probe cache: no DHT lookup, no probe fanout (#1165).

Source

pub fn probe_cache_miss(&self)

A cache-miss pull found no usable probe-cache entry and ran a fresh lookup + probe (#1165).

Source

pub fn probe_collection_latency(&self, elapsed: Duration)

One probe round’s collection window ended (ADR 001 §Probe response collection): elapsed runs from the start of the concurrent probes to the end of collection.

Source

pub fn node_pull_first_byte(&self, elapsed: Duration)

One node-to-node pull leg read its first bao bytes elapsed after its open started.

Source

pub fn dht_lookup_round_ceiling(&self)

A DHT lookup was truncated at MAX_LOOKUP_ROUNDS while still finding closer nodes (#1145 review). A sustained rate means the ceiling is too low for the network size.

Source

pub fn node_pull_corruption(&self)

An upstream served hash-mismatched bytes for a paid pull (#831).

Source

pub fn node_pull_unreachable(&self)

A probe or pull to a candidate failed at the transport (#831).

Source

pub fn node_region_latency_penalty(&self)

A probed peer claimed this node’s own region but exceeded the ADR 030 latency ceiling, so it took the local reputation penalty (#1177).

Source

pub fn node_pull_pool_open_failure(&self)

A buyer channel open/reuse failed before a pull could start (#831).

Source

pub fn node_pull_too_large(&self)

A selected upstream claimed a total_bytes above this node’s max_blob_size ceiling and the buyer rejected it before buffering (#840). A buyer-side policy decision, so it does not score the provider.

Source

pub fn node_pull_rate_above_ceiling(&self)

A selected upstream quoted a per-MB rate above this node’s effective buyer ceiling and the buyer refused before paying (#1375). A buyer-side policy decision, so it does not score the provider.

Source

pub fn serve_stream_rejected_evicted_since_probe(&self)

Record a paid-delivery (serve_stream) request refused because the blob was evicted between probe and stream (#876).

Source

pub fn serve_stream_rejected_cache_miss(&self)

Record a serve_stream request refused on a cache miss that pull-through could not fill (#876).

Source

pub fn serve_stream_rejected_internal_error(&self)

Record a serve_stream request refused by a local store fault, signed as Declined (#876).

Source

pub fn serve_stream_rejected_unknown_lane(&self)

Record a serve_stream request refused on an unknown lane (#876).

Source

pub fn serve_stream_rejected_owner_mismatch(&self)

Record a serve_stream request refused because the client binding did not authorize the named channel (#876).

Source

pub fn serve_stream_rejected_bad_binding(&self)

A client binding failed to verify and the stream was reset.

Source

pub fn probe_read_fault(&self)

A probe request could not be read.

Source

pub fn serve_stream_rejected_insufficient_deposit(&self)

Record a serve_stream cache-miss refused by the pre-flight deposit guard (#856): the requesting channel could not cover the worst-case blob cost, so no upstream pull was started.

Source

pub fn serve_stream_rejected_pool_unconfirmed(&self)

Record a serve_stream admission refused because a wired PoolView could not confirm the pool on-chain (absent, closed, or the admit getPool faulted), or a getAuthorization fault left the signer with no cached registered read — kept distinct from a real deposit-exhaustion refusal.

Source

pub fn serve_stream_rejected_pool_closing(&self)

Record a serve_stream admission refused because its pool is Closing.

Source

pub fn serve_signer_auth_cached(&self)

Record an admit signer confirm the signer-auth cache answered.

Source

pub fn serve_signer_auth_first_read(&self)

Record an admit signer confirm for a (pool, signer) with no held read.

Source

pub fn serve_signer_auth_reread(&self)

Record an admit signer confirm that re-reads an Unregistered signer.

Source

pub fn serve_stream_rejected_signer_cap_exhausted(&self)

Record a serve_stream admission refused because the request’s voucher signer has drained its shared on-chain cap (cap − spent below the serve floor) or its registration has expired — a capability this node could never cash.

Source

pub fn serve_stream_midstream_signer_cap_exhausted(&self)

Record a live serve stopped mid-stream because its voucher signer drained its shared on-chain cap at other nodes after admission, so the headroom the node holds on the lane no longer covers a serve floor.

Source

pub fn serve_stream_rejected_signer_floor_at_cap(&self)

Record a serve_stream delivery refused because this capability signer hit its per-signer live concurrency cap of un-vouchered reservation, while the pool as a whole can still pay.

Source

pub fn serve_stream_rejected_pull_loop_guard(&self)

Record a whole-blob request from an active staker refused while this node’s own upstream open for the blob was in progress (#2224).

Source

pub fn serve_stream_rejected_range_not_satisfiable(&self)

Record a serve_stream delivery refused because the requested bounded range is out of bounds for the blob (ADR 005 §Bounded byte ranges).

Source

pub fn serve_stream_rejected_blob_too_large(&self)

Record a serve_stream pull-through refused because the requested range starts at or past the node’s max_blob_size ceiling (ADR 005 §max_blob_size enforcement).

Source

pub fn serve_stream_rejected_hash_denied(&self)

Record a serve_stream delivery refused because the blob is on the operator’s local denylist (ADR 011 §Local Denylist).

Source

pub fn serve_stream_rejected_chain_hash_denied(&self)

Record a serve_stream delivery refused because the blob is on the governance blacklist (ADR 011 §On Blacklist Event).

Source

pub fn serve_stream_rejected_origin_denied(&self)

Record a serve_stream delivery refused because the channel’s funding address is a blacklisted origin (ADR 011 §On Blacklist Event).

Source

pub fn serve_stream_rejected_foreign_declined(&self)

Record a serve_stream delivery declined by the origin-only policy (#1759, #1766): a live probe of this node’s own backend genuinely does not hold the hash.

Source

pub fn serve_stream_rejected_chain_stale(&self)

Record a serve_stream request refused because the node’s chain reads are stale past blockchain.chain_staleness_grace_sec (ADR 011 § Serving while chain-stale).

Source

pub fn serve_stream_terminated_takedown(&self)

Record an in-flight delivery cut off at an MB boundary because a takedown landed after the stream opened (ADR 011 §On Blacklist Event).

Source

pub fn serve_stream_rejected_load_shed_hit(&self)

Record a serve_stream cache-hit request refused by the load-shed policy (egress saturation).

Source

pub fn serve_stream_rejected_load_shed_miss(&self)

Record a serve_stream cache-miss request refused by the load-shed policy (concurrency pressure or per-client fairness).

Source

pub fn serve_frame_accounting_fault(&self)

Record a serve leg refusing to cut or frame a ChunkData because its own byte accounting did not add up. Node-side bug, never peer behaviour.

Source

pub fn serve_stream_node_fault(&self)

Record a cdn/client/v1 delivery that ended on a node-side fault rather than on a peer hang-up or a client payment fault. Node-side bug, never peer behaviour.

Source

pub fn serve_stream_proof_budget_exhausted(&self)

Record a cdn/client/v1 delivery that ended because the payer spent its per-chunk proof budget without settling the chunk. The payer’s fault, not a node fault.

Source

pub fn serve_stream_rejected_stream_cap_full(&self)

Record a cdn/client/v1 stream reset because the connection’s stream cap was full.

Source

pub fn serve_stream_request_unreadable(&self)

Record a cdn/client/v1 stream reset because its first request could not be read. Peer-side.

Source

pub fn serve_stream_voucher_rejected(&self)

Record a paying cdn/client/v1 stream stopped on a rejected voucher.

Source

pub fn serve_stream_client_declined(&self)

Record a cdn/client/v1 stream the requester left before any voucher credited a byte. Peer-side.

Source

pub fn serve_stream_client_abandoned(&self)

Record a paying cdn/client/v1 stream the requester left after a voucher credited bytes. Peer-side.

Source

pub fn load_shed_egress_bps(&self, bps: i64)

Record the current measured egress EWMA, bytes/sec.

Source

pub fn load_shed_pressure_active(&self, active: bool)

Record whether the load-shed policy considers the node pressured (1 for yes, 0 for no).

Source

pub fn load_shed_streams_in_flight(&self, n: u32)

Publish node-wide serves in flight, sampled from the shed controller beside the egress EWMA. u32 -> i64 is lossless.

Source

pub fn warming_speculative_blocked(&self)

Record an above-floor warming buy downgraded to the amortized floor because the source’s per-source allowance was spent (ADR 041).

Source

pub fn warming_sources_blocked(&self, n: usize)

Publish how many upstream sources currently hold no warming allowance (ADR 041), sampled beside the egress EWMA.

Source

pub fn node_pull_through_window_paused(&self)

The window-paced serve loop paused the upstream pull at the ramped credit window to wait for the downstream voucher to clear (#856, #1669).

Source

pub fn node_pull_through_min_draw_waits(&self)

The window-paced serve loop paused the upstream pull until the ramped window opened to the minimum draw (#2061).

Source

pub fn node_pull_through_wait(&self, elapsed: Duration)

A window-paced pause of the upstream pull ended after elapsed.

Source

pub fn node_pull_through_client_abandoned(&self)

A window-paced serve was abandoned because the requesting client dropped or underpaid mid-pull (#856).

Source

pub fn node_pull_through_upstream_verify_failed(&self)

A serve-miss pull ingested upstream bytes that failed bao verification against the content root (#856/#915).

Source

pub fn local_outboard_serve(&self)

The node entered serve_via_backend_origin — the stream-while-store serve tier fired for this request (#1130). Counted at entry, before any admission guard; distinguishes this tier from the buffered populate_local fallback regardless of this request’s eventual outcome.

Source

pub fn serve_cache_hit(&self)

The blob-availability gate classified a paid serve as servable from a complete local blob. Bumped before the shed gate and before any fill tier can run.

Source

pub fn serve_cache_partial_hit(&self)

The gate classified a paid serve as servable from a partial blob whose held chunk groups cover the requested span (#1506).

Source

pub fn serve_cache_miss(&self)

The gate could not satisfy a paid serve locally, so a fill tier runs.

Source

pub fn node_pull_timeout(&self)

A buyer→upstream pull hit this node’s own pull_timeout deadline (#857). A buyer-side condition, so it does not score the provider’s reputation.

Source

pub fn node_pull_voucher_rejected(&self)

An upstream rejected a voucher this node presented mid-pull (#857) — a buyer payment-side fault, so it does not score the provider’s reputation.

Source

pub fn node_pull_pool_wedged(&self)

A lane can no longer pay but the pool’s deposit is still escrowed, so the row was KEPT for the reclaim sweep and the provider suppressed instead (#1145 review). Money at rest — see the counter’s docs.

Source

pub fn node_pull_refused(&self)

A selected upstream refused delivery up front (#1144). Counts every wire class; none scores the provider’s reputation.

Source

pub fn node_pull_refused_unattributable(&self)

A refusal this node cannot attribute to the upstream (#1520) — NotFound or Unfunded. A rate approaching node_pull_refused means nobody will serve us, which is usually our own deposit or binding, not their fault.

Source

pub fn node_upstream_rate_limited(&self)

An upstream shed a probe or pull at the transport with APP_ERR_RATE_LIMITED (#1986). Suppressed briefly, never scored — see the counter’s docs.

Source

pub fn node_pull_backpressure_backoffs(&self)

A ranged run completed one wait on an upstream’s backpressure refusal instead of dropping the sole covering source (#2178) — see the counter’s docs.

Source

pub fn node_pull_backpressure_exhausted(&self)

The sole covering source outlasted the backpressure wait budget and the assembly ended short (#2178) — see the counter’s docs.

Source

pub fn node_pull_leg_no_progress(&self)

A clean upstream leg moved neither frontier and the run dropped its source (#2194) — see the counter’s docs.

Source

pub fn node_pull_stalled(&self)

An upstream went silent mid-stream (#1134); the pull was abandoned and the provider scored Unreachable.

Source

pub fn node_pull_local_fault(&self)

A pull failed for a LOCAL reason (#1145 review) — signer, encode, range, or this node’s own cache store — so the upstream was exonerated. Says nothing about the network; any sustained rate means this node cannot complete a pull.

Source

pub fn node_pull_pool_open_pending(&self)

A buyer channel open outlived the per-candidate budget (#1143). The open continues in the background; the pull moves on. No reputation effect.

Source

pub fn node_pull_progress_persist_failure(&self)

A pull paid ≥1 voucher but persisting the buyer channel resume watermark failed (#852); the channel’s stored progress now lags the upstream.

Source

pub fn node_pull_progress_dropped(&self)

A paid voucher’s watermark was DROPPED because the provider’s channel slot had been replaced by a newer open before the write landed (#1145 review).

Source

pub fn node_pull_progress_superseded(&self)

A concurrent settle on the shared channel ledger persisted a higher watermark first, so this write was superseded (benign under BuyerLedgers; #1145 review).

Source

pub fn node_pull_recovery_step(&self)

A miss fill’s funding recovery step funded the next pass (#1530).

Source

pub fn node_pull_recovery_step_refused(&self)

A miss fill’s funding recovery step had no way to fund the next pass, or its funding tx failed (#1530).

Source

pub fn node_pull_through_timeout(&self)

The delivery handler abandoned a pull-through at its deadline (#831).

Source

pub fn node_pull_through_error(&self)

A cache-engine error (not a clean miss) was hit filling a miss (#831).

Source

pub fn origin_directory_get_origins_failure(&self)

A getOrigins lookup failed on a cold-namespace cache miss in the lazy origin directory. The lookup fails closed (resolves no origins for that request) and the failure is not cached, so this is the drift signal to alert on.

Source

pub fn origin_directory_cache_size(&self, count: usize)

Publish the current namespace count held in the lazy origin directory cache (positive + negative entries), after an insert.

Source

pub fn connection_opened(&self)

A connection was accepted and handed to a handler.

Source

pub fn connection_closed(&self)

A connection closed. Pairs with connection_opened.

Source

pub fn client_idle_close(&self)

A cdn/client/v1 connection was reaped by the application-layer idle closer (ADR 005 §Connection lifetime).

Source

pub fn otlp_export_failure(&self)

An OTLP span-export batch failed (commands::otlp’s counting exporter).

Source

pub fn rpc_healthy(&self, ok: bool)

Set the RPC health gauge. true -> 1 (reachable), false -> 0 (unreachable). Driven by the watchdog task spawned in runtime::run.

Source

pub fn dispatch_rejected_global(&self)

Record a connection rejected by the global concurrency semaphore.

Source

pub fn dispatch_rejected_per_source(&self)

Record a connection rejected by the per-source rate limiter.

Source

pub fn dispatch_permit_acquired(&self)

Increment the in-flight dispatch permit gauge.

Source

pub fn dispatch_permit_released(&self)

Decrement the in-flight dispatch permit gauge.

Source

pub fn dispatch_per_source_skipped_no_addr(&self)

Record a relay-only connection accepted while the per-source layer was enabled but no peer IP could be resolved at accept time.

Source

pub fn inbound_connection_direct(&self)

Record an inbound connection that arrived with a direct IP path.

Source

pub fn inbound_connection_relayed(&self)

Record an inbound connection that arrived over the relay only.

Source

pub fn node_public_address(&self, public: bool)

Set whether the host has a public address on its default route.

Source

pub fn dht_rate_limit_rejected_per_peer(&self)

Record a cdn/dht/v1 request rejected at the per-peer layer.

Source

pub fn dht_rate_limit_rejected_per_ip(&self)

Record a cdn/dht/v1 request rejected at the per-IP layer.

Source

pub fn dht_rate_limit_rejected_global(&self)

Record a cdn/dht/v1 request rejected at the global layer.

Source

pub fn dht_rate_limit_prune_sweep_per_ip(&self)

Record a retain_recent sweep of the per-IP DHT keyed-limiter map (#645).

Source

pub fn dht_rate_limit_prune_sweep_per_peer(&self)

Record a retain_recent sweep of the per-peer DHT keyed-limiter map (#645).

Source

pub fn dht_rate_limit_tracked_per_ip_set(&self, n: usize)

Set the per-IP DHT keyed-limiter tracked-size gauge (#645). The try_from(...).unwrap_or(i64::MAX) clamp matches the existing gauge-set pattern elsewhere in this module and stays within the workspace’s anti-panic policy.

Source

pub fn dht_rate_limit_tracked_per_peer_set(&self, n: usize)

Set the per-peer DHT keyed-limiter tracked-size gauge (#645).

Source

pub fn probe_rate_limit_rejected_per_peer(&self)

Record a cdn/probe/v1 request rejected at the per-peer layer.

Source

pub fn probe_rate_limit_rejected_per_ip(&self)

Record a cdn/probe/v1 request rejected at the per-IP layer.

Source

pub fn probe_rate_limit_rejected_global(&self)

Record a cdn/probe/v1 request rejected at the global layer.

Source

pub fn probe_rate_limit_prune_sweep_per_ip(&self)

Record a retain_recent sweep of the per-IP probe keyed-limiter map (#645).

Source

pub fn probe_rate_limit_prune_sweep_per_peer(&self)

Record a retain_recent sweep of the per-peer probe keyed-limiter map (#645).

Source

pub fn probe_rate_limit_tracked_per_ip_set(&self, n: usize)

Set the per-IP probe keyed-limiter tracked-size gauge (#645).

Source

pub fn probe_rate_limit_tracked_per_peer_set(&self, n: usize)

Set the per-peer probe keyed-limiter tracked-size gauge (#645).

Source

pub fn dht_request_failed(&self)

Record a cdn/dht/v1 request that was admitted by the rate limiter but failed after that (frame decode, write, encode, timeout, etc).

Source

pub fn dht_store_rejected_holder_mismatch(&self)

Record a Store rejected by the holder != authenticated NodeId check (ADR 022 §STORE Flow line 140 — lying-holder attack).

Source

pub fn dht_store_rejected_non_staked(&self)

Record a Store rejected by the active-staker filter (ADR 022 §STORE Flow line 140 — non-staked publisher).

Source

pub fn dht_store_rejected_quota(&self)

Record a Store rejected by the per-publisher quota (ADR 022 §Content Records and TTL — 200-record hard cap).

Source

pub fn dht_store_accepted(&self)

Record a Store admitted to the record store (newly inserted or refreshed).

Source

pub fn dht_batch_store_received(&self)

Record a BatchStore that reached per-hash admission (passed stage-1 rate limiting + the batch-level holder check, #648).

Source

pub fn dht_batch_store_hashes_deferred_rate_limit(&self, count: u64)

Record count BatchStore hashes deferred (acked false) because the two-stage rate-limit budget ran out before reaching them (ADR 022 §Batch token accounting, #648).

Source

pub fn dht_republish_lag_sweep(&self)

Record a republish lag sweep starting (the cache-commit broadcast channel reported Lagged).

Source

pub fn dht_republish_lag_sweep_coalesced(&self)

Record a lag folded into an already-running or already-queued sweep rather than starting a walk of its own.

Source

pub fn dht_republish_lag_sweep_panicked(&self)

Record a lag-sweep walk that died in a panic instead of completing.

Source

pub fn dht_republish_sweep_reseeded(&self, count: u64)

Record count hashes a lag sweep newly scheduled for republish.

Source

pub fn dht_republish_seed_store_walk_failure(&self)

Record a bulk republish seed (boot cold start or lag sweep) that could not walk the store and covered only the origin-held half.

Source

pub fn dht_republish_seed_origin_probe_failures(&self, count: u64)

Record count stored hashes a bulk republish seed could not put to its origin, leaving them out of the announce set.

Source

pub fn slash_watcher_tick(&self)

Stamp the slash watcher’s *_last_tick_timestamp_seconds liveness gauge with the current wall-clock time (#1316). The on_tick_success hook, fired on EVERY successful poll tick so a dead task’s gauge goes stale.

Source

pub fn chain_get_logs_span(&self, span: u64)

Record the chain-event poller’s current eth_getLogs window span.

Source

pub fn chain_get_logs_range_rejected(&self)

Count one eth_getLogs window the provider rejected for its range.

Source

pub fn chain_get_logs_retried(&self)

Count one in-tick retry of a transiently failed eth_getLogs window.

Source

pub fn chain_get_logs_deferred(&self)

Count one eth_getLogs window deferred to the next tick.

Source

pub fn chain_boot_read_retried(&self)

Count one boot-time chain read retried after a transient error.

Source

pub fn staker_set_watcher_tick(&self)

Stamp the staker-set watcher’s liveness gauge (#1316). See slash_watcher_tick.

Source

pub fn capacity_bond_registry_resync_failure(&self)

A capacity-bond registry re-enumeration failed and the previous projections were kept. Bumps capacity_bond_registry_resync_failures_total. Pairs with the warn! in crate::dht::capacity_bond_registry’s RegistrySink::on_tick_complete.

Source

pub fn capacity_bond_registry_resync(&self)

Stamp the capacity-bond registry’s re-enumeration liveness gauge on a successful resync. Reads 0 until the first one lands, so any alert on it needs a > 0 guard.

Source

pub fn blacklist_watcher_tick(&self)

Stamp the blacklist watcher’s liveness gauge (#1316, #1320).

Source

pub fn settlement_watcher_tick(&self)

Stamp the payment-settlement watcher’s liveness gauge (#1316).

Source

pub fn fee_shares_watcher_tick(&self)

Stamp the fee-shares watcher’s liveness gauge. See slash_watcher_tick; this one matters because the fee-shares watcher’s failure paths deliberately return Ok, so this gauge is the only signal that it is still polling.

Source

pub fn slash_watcher_task_panicked(&self)

Record that the slash watcher task unwound on a panic (#1316). Bumped from the Drop guard in multiplexed_poller::run via the on_task_panic hook.

Source

pub fn staker_set_watcher_task_panicked(&self)

Record that the staker-set watcher task unwound on a panic (#1316).

Source

pub fn blacklist_watcher_task_panicked(&self)

Record that the blacklist watcher task unwound on a panic (#1316, #1283).

Source

pub fn fee_shares_watcher_task_panicked(&self)

Record that the fee-shares watcher task unwound on a panic.

Source

pub fn settlement_watcher_task_panicked(&self)

Record that the payment-settlement watcher task unwound on a panic (#1316).

Source

pub fn blacklist_enforcement_failure(&self, count: u64)

Record count hashes a blacklist re-scope could not enforce this pass (#1319 — Recheck::Failed). One aggregated bump per pass, not per hash.

Source

pub fn blacklist_reenumeration_failure(&self)

A blacklist periodic re-enumeration failed and the current deny-set was kept. Bumps blacklist_reenumeration_failures_total.

Source

pub fn bytes_served(&self, bytes: u64)

Count bytes of payload written to a client on cdn/client/v1.

Source

pub fn bytes_received(&self, bytes: u64)

Count bytes of payload admitted from an upstream node in a paid pull.

Source

pub fn pool_redemptions(&self, vouchers: u64)

A landed redeemMany redeemed vouchers vouchers on-chain.

Source

pub fn voucher_received(&self)

An inbound stream accepted a voucher as the payee.

Source

pub fn preimage_reveal_received(&self)

An inbound stream accepted a PayWord preimage reveal that advanced its lane’s chain frontier.

Source

pub fn pool_grace_close(&self)

A pool entered the owner-close grace window while this node held unredeemed vouchers on it.

Source

pub fn onchain_tx_landed(&self)

A node transaction mined and succeeded.

Source

pub fn onchain_tx_reverted(&self)

A node transaction mined and reverted.

Source

pub fn onchain_tx_send_failed(&self)

The RPC refused a node transaction at send.

Source

pub fn onchain_tx_receipt_failed(&self)

An issued node transaction’s receipt wait failed and no by-hash fetch found the receipt.

Source

pub fn onchain_tx_timeout(&self)

An issued node transaction’s receipt did not arrive in time and no by-hash fetch found it.

Source

pub fn onchain_tx_receipt_recovered(&self)

A failed or lapsed receipt wait was resolved by fetching the receipt by hash.

Source

pub fn dht_findvalue_query(&self)

A FIND_VALUE provider lookup started.

Source

pub fn dht_lookup_round_timeout(&self)

A lookup round hit its timeout and aborted its in-flight RPCs.

Source

pub fn dht_store_published(&self, count: u64)

count peers accepted a STORE record from this node.

Source

pub fn dht_routing_table_size(&self, entries: usize)

Publish the routing table’s entry count.

Source

pub fn dht_bucket_refresh_failure(&self)

A bucket refresh failed.

Source

pub fn dht_bootstrap_find_node_failures(&self, count: u64)

count bootstrap FIND_NODE RPCs against seeds failed.

Source

pub fn fee_shares_watcher_poll_failure(&self)

The fee-shares watcher’s authoritative getShares() re-read failed.

Source

pub fn fee_shares_watcher_unregistered(&self, unregistered: bool)

Record whether the fee-shares watcher is unregistered because the startup feeRouter() read failed.

Source

pub fn config_reload_failure(&self)

A config reload failed and kept the previous values.

Source

pub fn receipt_write_failure(&self)

The receipt writer failed to persist one audit record.

Source

pub fn serve_stream_midstream_pool_exhausted(&self)

A paying stream stopped mid-delivery on PoolExhausted.

Source§

impl Metrics

Source

pub fn slash_watcher_backoff_started(&self)

The slash-detection watcher’s cycle errored and the loop is about to back off (#1032). Stamps slash_watcher_down_since (once per drift window) so slash_watcher_down_seconds climbs until the next healthy cycle. Mirrors Self::staker_set_watcher_backoff_started; a poisoned lock skips the update (the gauge then reads i64::MAX at scrape — the safe alerting direction).

Source

pub fn slash_watcher_cycle_established(&self)

Mark the slash-detection watcher cycle established (#1032): clear slash_watcher_down_since so slash_watcher_down_seconds reads 0 for the life of the cycle. Mirrors Self::staker_set_watcher_cycle_established.

Source

pub fn staker_set_watcher_backoff_started(&self)

A staker-set watcher poll tick failed with an error and the loop is about to back off (#783, crate::dht::chain_staker_set). Stamps down_since so staker_set_watcher_down_seconds begins to climb, and — on the transition from a healthy cycle into the error state (down_since was None) — bumps staker_set_watcher_restarts_total exactly once per drift window, rather than once per backoff iteration of one continuous outage. A poisoned lock is treated as “skip the update” rather than panicking (anti-panic policy); the gauge’s poison fallback (i64::MAX) still keeps the alert tripped. This method is the on_backoff hook wired in crate::dht::chain_staker_set; it pairs with the loop-level warn! in multiplexed_poller::run ("watcher RPC error; restarting after backoff").

Source

pub fn staker_set_watcher_cycle_established(&self)

Mark the staker-set watcher’s poll cycle as established (#783, downtime semantics #788): a poll tick succeeded and logs are flowing again (the on_established hook). Clears down_since to None so staker_set_watcher_down_seconds reads 0 for the entire life of this cycle, however long. A poisoned lock is treated as “skip the update” rather than panicking (anti-panic policy); the gauge’s poison fallback (i64::MAX) then keeps the alert tripped — the safe (alerting) direction.

Source

pub fn blacklist_watcher_backoff_started(&self)

The blacklist watcher’s poll tick errored and the loop is about to back off (#1283). Stamps blacklist_watcher_down_since once per drift window so blacklist_watcher_down_seconds climbs until the next healthy cycle. Tracks chain-read outages only; an enforcement failure that leaves a blacklisted blob servable is the separate blacklist_enforcement_failures counter (#1319). Mirrors Self::slash_watcher_backoff_started.

Source

pub fn blacklist_watcher_cycle_established(&self)

Mark the blacklist watcher cycle established (#1283): clear blacklist_watcher_down_since so blacklist_watcher_down_seconds reads 0 for the life of the cycle. Mirrors Self::slash_watcher_cycle_established.

Source

pub fn settlement_watcher_backoff_started(&self)

The payment-settlement watcher’s poll tick errored and the loop is about to back off (#1316). Stamps settlement_watcher_down_since once per drift window. Mirrors Self::slash_watcher_backoff_started.

Source

pub fn settlement_watcher_cycle_established(&self)

Mark the payment-settlement watcher cycle established (#1316): clear settlement_watcher_down_since. Mirrors Self::slash_watcher_cycle_established.

Source

pub fn fee_shares_watcher_backoff_started(&self)

The fee-shares watcher’s poll tick errored and the loop is about to back off. Stamps fee_shares_watcher_down_since once per drift window. Mirrors Self::slash_watcher_backoff_started.

Source

pub fn fee_shares_watcher_cycle_established(&self)

Mark the fee-shares watcher cycle established: clear fee_shares_watcher_down_since. Mirrors Self::slash_watcher_cycle_established.

Trait Implementations§

Source§

impl Debug for Metrics

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for Metrics

Source§

fn default() -> Self

Returns the “default value” for a type. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T> FutureExt for T

Source§

fn with_context(self, otel_cx: Context) -> WithContext<Self> ⓘ

Attaches the provided Context to this type, returning a WithContext wrapper. Read more
Source§

fn with_current_context(self) -> WithContext<Self> ⓘ

Attaches the current Context to this type, returning a WithContext wrapper. Read more
Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<T> IntoRequest<T> for T

Source§

fn into_request(self) -> Request<T>

Wrap the input message T in a tonic::Request
Source§

impl<Unshared, Shared> IntoShared<Shared> for Unshared
where Shared: FromUnshared<Unshared>,

Source§

fn into_shared(self) -> Shared

Creates a shared type from an unshared type.
Source§

impl<L> LayerExt<L> for L

Source§

fn named_layer<S>(&self, service: S) -> Layered<<L as Layer<S>>::Service, S>
where L: Layer<S>,

Applies the layer to a service and wraps it in Layered.
Source§

impl<T> MaybeSend for T
where T: Send,

Source§

impl<T> PolicyExt for T
where T: ?Sized,

Source§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow only if self and other return Action::Follow. Read more
Source§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow if either self or other returns Action::Follow. Read more
Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V

Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more