Struct AwaMetrics 

pub struct AwaMetrics {

    pub jobs_inserted: Counter<u64>,
    pub enqueue_batch_size: Histogram<u64>,
    pub enqueue_duration_seconds: Histogram<f64>,
    pub jobs_completed: Counter<u64>,
    pub jobs_failed: Counter<u64>,
    pub jobs_retried: Counter<u64>,
    pub jobs_cancelled: Counter<u64>,
    pub jobs_claimed: Counter<u64>,
    pub claim_batches: Counter<u64>,
    pub dispatch_wakeups: Counter<u64>,
    pub dispatch_wake_to_claim_seconds: Histogram<f64>,
    pub dispatch_capacity_available: Histogram<u64>,
    pub dispatch_empty_claims: Counter<u64>,
    pub dispatch_unused_permits: Counter<u64>,
    pub dispatch_rate_limited: Counter<u64>,
    pub claim_batch_size: Histogram<u64>,
    pub claim_duration_seconds: Histogram<f64>,
    pub job_duration_seconds: Histogram<f64>,
    pub completion_flushes: Counter<u64>,
    pub completion_flush_batch_size: Histogram<u64>,
    pub completion_flush_duration_seconds: Histogram<f64>,
    pub promotion_batches: Counter<u64>,
    pub promotion_batch_size: Histogram<u64>,
    pub promotion_duration_seconds: Histogram<f64>,
    pub jobs_in_flight: UpDownCounter<i64>,
    pub heartbeat_batches: Counter<u64>,
    pub maintenance_rescues: Counter<u64>,
    pub jobs_waiting_external: Counter<u64>,
    pub queue_depth: Gauge<i64>,
    pub queue_lag_seconds: Gauge<f64>,
    pub wait_duration_seconds: Histogram<f64>,
    pub dlq_moved: Counter<u64>,
    pub dlq_retried: Counter<u64>,
    pub dlq_purged: Counter<u64>,
    pub dlq_depth: Gauge<i64>,
    pub queue_info: Gauge<i64>,
    pub job_kind_info: Gauge<i64>,
    pub storage_transition_ready: Gauge<i64>,
    pub storage_canonical_live_backlog: Gauge<i64>,
    pub storage_live_runtime_capability: Gauge<i64>,
    pub storage_state: Gauge<i64>,
    pub maintenance_rotate_attempts: Counter<u64>,
    pub maintenance_rotate_skipped_rows: Histogram<u64>,
    pub maintenance_prune_attempts: Counter<u64>,
    pub maintenance_prune_skipped_rows: Histogram<u64>,
    pub maintenance_branch_duration_seconds: Histogram<f64>,
    pub maintenance_branch_overrun_total: Counter<u64>,
    pub ring_current_slot: Gauge<i64>,
    pub ring_generation: Gauge<i64>,
}

Awa worker metrics backed by OpenTelemetry.

Fields

jobs_inserted: Counter<u64>

Total jobs inserted.

enqueue_batch_size: Histogram<u64>

Per-batch size distribution for the direct queue-storage COPY enqueue path (QueueStorage::enqueue_params_copy / Python Client.enqueue_many_copy). Lets producers see how chunky their batches actually land — useful when a target enqueue rate isn’t being hit and you need to disambiguate “batches are tiny” from “batches are slow.”

enqueue_duration_seconds: Histogram<f64>

Per-batch wall-clock duration for the direct queue-storage COPY enqueue path. Pair with enqueue_batch_size to read jobs-per-second and per-batch latency in one Grafana row.

jobs_completed: Counter<u64>

Total jobs completed successfully.

jobs_failed: Counter<u64>

Total jobs that failed (terminal).

jobs_retried: Counter<u64>

Total jobs marked retryable.

jobs_cancelled: Counter<u64>

Total jobs cancelled.

jobs_claimed: Counter<u64>

Total jobs claimed (dequeued) for execution.

claim_batches: Counter<u64>

Number of dispatcher claim queries executed.

dispatch_wakeups: Counter<u64>

Number of dispatcher wake-ups by reason.

dispatch_wake_to_claim_seconds: Histogram<f64>

Time from wake-up to the first claim attempt.

dispatch_capacity_available: Histogram<u64>

Number of permits available when a dispatcher wake is processed.

dispatch_empty_claims: Counter<u64>

Number of wakes that found no jobs despite available capacity.

dispatch_unused_permits: Counter<u64>

Number of pre-acquired permits released unused after a claim round.

dispatch_rate_limited: Counter<u64>

Number of wakes that were blocked by rate limiting.

claim_batch_size: Histogram<u64>

Claim batch size distribution.

claim_duration_seconds: Histogram<f64>

Claim query duration.

job_duration_seconds: Histogram<f64>

Job execution duration.

completion_flushes: Counter<u64>

Number of completion batch flushes executed.

completion_flush_batch_size: Histogram<u64>

Completion flush batch size distribution.

completion_flush_duration_seconds: Histogram<f64>

Completion flush duration.

promotion_batches: Counter<u64>

Number of scheduled/retryable promotion batches executed.

promotion_batch_size: Histogram<u64>

Promotion batch size distribution.

promotion_duration_seconds: Histogram<f64>

Promotion query duration.

jobs_in_flight: UpDownCounter<i64>

Current in-flight jobs (can go up and down).

heartbeat_batches: Counter<u64>

Total heartbeat batches sent.

maintenance_rescues: Counter<u64>

Total maintenance rescue operations.

jobs_waiting_external: Counter<u64>

Total jobs parked for external callback.

queue_depth: Gauge<i64>

Current queue depth per state — how many jobs are in each state per queue.

queue_lag_seconds: Gauge<f64>

Queue lag — age of the oldest available job per queue.

wait_duration_seconds: Histogram<f64>

Time from job creation to claim — the user-visible queuing latency.

dlq_moved: Counter<u64>

Total jobs moved into the Dead Letter Queue.

dlq_retried: Counter<u64>

Total jobs retried out of the Dead Letter Queue.

dlq_purged: Counter<u64>

Total DLQ rows purged.

dlq_depth: Gauge<i64>

Current DLQ depth per queue.

queue_info: Gauge<i64>

Info gauge for declared queue descriptors — value is always 1, the useful payload is the attribute set (display_name, owner, tags). Dashboards join it into throughput / latency panels with a * on(awa_job_queue) group_left(awa_queue_display_name, awa_queue_owner) Prometheus expression, which keeps descriptor fields out of the high-cardinality per-metric label set.

job_kind_info: Gauge<i64>

Info gauge for declared job-kind descriptors. Same pattern as queue_info.

storage_transition_ready: Gauge<i64>

Readiness gauge for storage transition actions such as enter_mixed_transition and finalize (1 = ready, 0 = blocked).

storage_canonical_live_backlog: Gauge<i64>

Current canonical live backlog observed by queue-storage-capable runtimes.

storage_live_runtime_capability: Gauge<i64>

Current live runtime count per reported storage capability.

storage_state: Gauge<i64>

One-hot info gauge for the current storage transition state and engines.

maintenance_rotate_attempts: Counter<u64>

Maintenance ring rotation attempts, attributed to (ring, outcome, blocker). For Rotated outcomes the awa.ring.blocker attribute is “none”; for SkippedBusy it carries the per-ring blocker label (“queue.ready_rows”, “queue.claim_attempt_batches”, “queue.done_rows”, “queue.tombstone_rows”, “queue.ready_segments”, “queue.receipt_completion_batch_rows”, “queue.receipt_completion_tombstone_rows”, “queue.terminal_delta_rows”, “lease.rows”, “claim.rows”, “claim.closure_rows”, “claim.closure_batch_rows”). One increment per non-zero blocker means a single SkippedBusy with multiple populated fields emits multiple events; that’s intentional so dashboards can attribute blame independently.

maintenance_rotate_skipped_rows: Histogram<u64>

Magnitudes of the per-blocker row counts when a rotation is SkippedBusy. Histogram so dashboards can show whether the ring is pinned by handfuls of stragglers or by mountains of unfinished work.

maintenance_prune_attempts: Counter<u64>

Maintenance ring prune attempts, attributed to (ring, outcome, reason). reason=“none” for Pruned/Noop/Blocked; otherwise carries the SkipReason discriminator (e.g. “queue.active_leases”, “queue.pending_ready”, “claim.open”).

maintenance_prune_skipped_rows: Histogram<u64>

Magnitude of the reason count when prune returns SkippedActive.

maintenance_branch_duration_seconds: Histogram<f64>

Per-branch wall-clock duration for the maintenance leader’s main tokio::select! loop, attributed by awa.maintenance.branch. Records the time from the moment a select arm fires to the moment its body returns — so each sample is the body’s contribution to head-of-line delay for every other branch on the same loop. Dashboards alert on histogram_quantile(0.99, ...) per branch to see whether any one branch is dominating select-loop time. Issue #242.

maintenance_branch_overrun_total: Counter<u64>

Counter of “delayed tick” transitions per maintenance branch. One increment is emitted when a branch fires after running longer than its own tick interval in the previous iteration — i.e. the timer was already overdue at the moment it fired. Emits on transition (on-time -> delayed) only, not on every subsequent overrun tick, so a sustained slow branch produces one event per “overrun episode” rather than one per tick. Fleets alert on this rather than scraping the matching tracing::warn! line. Issue #242.

ring_current_slot: Gauge<i64>

Current ring current_slot per ring, sampled from each rotate call. The slot number itself isn’t meaningful but the rate of advance is — dashboards plot rate(slot_changes) to see whether rotation is healthy or pinned.

ring_generation: Gauge<i64>

Current ring generation per ring. Always-increasing; dashboards show its derivative as “rotations per minute”.

Implementations

impl AwaMetrics

pub fn new(meter: &Meter) -> AwaMetrics

Create metrics from an OpenTelemetry meter.

Instrument names come from names so the public constants and the registered instruments can’t drift — a rename in names::* updates the registration too.

pub fn from_global() -> AwaMetrics

Create metrics using the global OTel meter provider with meter name “awa”.

pub fn record_job_completed(&self, kind: &str, queue: &str, duration: Duration)

Record a job completion with duration and attributes.

pub fn record_job_failed(&self, kind: &str, queue: &str, terminal: bool)

Record a job failure.

pub fn record_job_retried(&self, kind: &str, queue: &str)

Record a job retry.

pub fn record_enqueue_batch( &self, queue: &str, batch_size: u64, duration: Duration, )

Record a producer batch through the direct queue-storage COPY enqueue path (QueueStorage::enqueue_params_copy / Python Client.enqueue_many_copy).

batch_size is the row count that COPY actually wrote; duration is the wall-clock time the producer spent in the COPY call (start to commit), not including pre-batch row preparation.

At batch_size = 0 this is a no-op — empty batches are valid callers and don’t need a metric sample.

pub fn record_job_claimed(&self, queue: &str, batch_size: u64)

Record a job claimed from queue.

Use this for the canonical engine, which has no per-shard concept. The queue-storage engine uses record_job_claimed_by_shard so dashboards can read per-shard fairness directly.

pub fn record_job_claimed_by_shard( &self, queue: &str, enqueue_shard: i16, batch_size: u64, )

Record a job claimed from queue, decorated with the enqueue shard.

Used by the queue-storage path so dashboards can sum awa.job.claimed by awa.enqueue.shard and confirm the claim ordering is rotating across shards rather than starving the higher-numbered ones. At enqueue_shards > 1 this is the only fairness signal that the operator gets from telemetry alone; at enqueue_shards = 1 the attribute is always 0 and the series is identical to the un-decorated form.

Call sites must not double-emit — invoke either this OR record_job_claimed, never both for the same claim, or the awa.job.claimed total will count each claim twice when dashboards sum across all attribute combinations.

pub fn record_claim_batch( &self, queue: &str, batch_size: u64, duration: Duration, )

Record a dispatcher claim query batch and its latency.

pub fn record_dispatch_wake(&self, queue: &str, reason: &str)

Record a dispatcher wake-up reason.

pub fn record_dispatch_wake_to_claim( &self, queue: &str, reason: &str, duration: Duration, )

Record time from wake-up to the first claim attempt.

pub fn record_dispatch_capacity_available( &self, queue: &str, reason: &str, permits: u64, )

Record how many permits were available on a dispatcher wake.

pub fn record_dispatch_empty_claim(&self, queue: &str, reason: &str)

Record a dispatcher wake that found no jobs.

pub fn record_dispatch_unused_permits(&self, queue: &str, count: u64)

Record permits released unused after a claim round.

pub fn record_dispatch_rate_limited(&self, queue: &str, reason: &str)

Record a wake that could not claim because of rate limiting.

pub fn record_completion_flush( &self, shard: usize, batch_size: u64, duration: Duration, )

Record a completion batch flush.

pub fn record_promotion_batch( &self, state: &str, batch_size: u64, duration: Duration, )

Record a scheduled/retryable promotion batch.

pub fn record_in_flight_change(&self, queue: &str, delta: i64)

Record in-flight change.

pub fn record_queue_depth(&self, queue: &str, state: &str, count: i64)

Record queue depth for a specific state.

pub fn record_queue_lag(&self, queue: &str, lag_seconds: f64)

Record queue lag (age of oldest available job).

pub fn record_wait_duration(&self, queue: &str, seconds: f64)

Record job wait duration (time from creation to claim).

pub fn record_dlq_moved(&self, kind: &str, queue: &str, reason: &str)

Record a job moved into the DLQ.

pub fn record_dlq_moved_bulk( &self, kind: Option<&str>, queue: Option<&str>, reason: &str, count: u64, )

Record a bulk admin move into the DLQ.

pub fn record_dlq_retried(&self, queue: Option<&str>, count: u64)

Record jobs retried out of the DLQ.

pub fn record_dlq_purged(&self, queue: Option<&str>, count: u64)

Record DLQ rows purged.

pub fn record_dlq_depth(&self, queue: &str, count: i64)

Record current DLQ depth for a queue.

pub fn record_queue_info( &self, queue: &str, display_name: Option<&str>, description: Option<&str>, owner: Option<&str>, docs_url: Option<&str>, tags: &[String], )

Emit the info gauge for a declared queue descriptor. Called once per descriptor on every runtime snapshot tick — constant value of 1 with the descriptor fields as attributes. Optional fields that are None are elided so we don’t produce display_name="" series.

pub fn record_job_kind_info( &self, kind: &str, display_name: Option<&str>, description: Option<&str>, owner: Option<&str>, docs_url: Option<&str>, tags: &[String], )

Emit the info gauge for a declared job-kind descriptor. Same shape as record_queue_info.

pub fn record_storage_transition_ready(&self, action: &str, ready: bool)

Record whether a storage transition action is currently ready.

pub fn record_storage_canonical_live_backlog(&self, count: i64)

Record canonical live backlog for the storage transition.

pub fn record_storage_live_runtime_capability( &self, capability: &str, count: i64, )

Record the number of live runtimes reporting a given storage capability.

pub fn record_storage_state(&self, status: &StorageStatus)

Emit the current storage transition state as a one-hot info gauge.

pub fn record_rotate_outcome(&self, ring: &'static str, outcome: &RotateOutcome)

Record a ring rotation outcome.

ring is one of “queue” / “lease” / “claim” — set by the caller based on which rotate fn returned the outcome. For SkippedBusy, every non-zero blocker count emits its own counter increment plus a histogram sample, so a queue rotate skipped on (ready=42, done=17) produces two events with different awa.ring.blocker labels. This makes sum by (awa.ring.blocker) (rate(...)) work cleanly in Grafana.

pub fn record_maintenance_branch_duration( &self, branch: &'static str, duration: Duration, )

Record the wall-clock duration of one maintenance tokio::select! arm. branch is a static name (e.g. "promote_scheduled", "rescue_stale_heartbeats") so the attribute set stays bounded. Issue #242.

pub fn record_maintenance_branch_overrun(&self, branch: &'static str)

Record one maintenance branch overrun episode — a transition from “on-time” to “delayed” for branch. Increments awa.maintenance.branch.overrun (Prometheus: awa_maintenance_branch_overrun_total{branch="<name>"}). Issue #242.

pub fn record_prune_outcome(&self, ring: &'static str, outcome: &PruneOutcome)

Record a ring prune outcome.

Trait Implementations

impl Clone for AwaMetrics

fn clone(&self) -> AwaMetrics

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for AwaMetrics

No-op metrics for when OTel is not configured.

fn default() -> AwaMetrics

Returns the “default value” for a type.