Skip to main content

Module metrics

Module metrics 

Source
Expand description

Process-local counters → Prometheus text. RFC 0010 §metrics. [feature: metrics]

Off the default path: the public record_* fns are no-ops unless built with --features metrics, so call sites stay clean and the default build pays nothing (metrics are otherwise derivable from the JSON-lines event stream — that is the default story). With the feature, a tiny dependency-free atomic registry backs an opt-in HTTP /metrics scrape surface (obs::serve).

Counters are per supervisor process. The long-lived root daemon’s surface reflects the runs it supervises — every one-shot, reaction, and scheduled fire flows through supervise_once — plus the tokens its direct children report up the control channel. Nested subagents keep their own (process-local) counters, still visible in their logs; cross-process metric rollup is a deliberate non-goal for v1 (same boundary as the tree token ceiling).

§The frozen metrics_schema contract (RFC 0016 §4)

RFC 0016 §4 freezes the metric names and label keys below into a versioned public API (METRICS_SCHEMA) a control plane (agentctl) authors dashboards/alerts/scalers against. The exposition mechanism is unchanged from RFC 0010 §3.8 (hand-written Prometheus 0.0.4 text — no prometheus/metrics crate). The §4.3 enumerated set is the contract; it is additive within the major and a removal/rename bumps the major (§8.2).

Cardinality (RFC 0016 §4.2, binding): /metrics is unauthenticated and may be bound on all interfaces. Labels carry bounded values only (status, model, type, server, tool, reason, limit, signal, phase, ok); never run_id / agent_id / agent_path / call_id / a resource URI — those are unbounded and live in logs/traces only. A control plane that needs per-run granularity reads the run report (§6) or the event stream (§7), never a metric. This module therefore stores label-bearing series as small fixed-domain atomic arrays (the closed label set is known at compile time), so the cardinality is structurally bounded.

Telemetry never crashes the agent (RFC 0016 §8.4): every fn here is a plain atomic add/store that cannot fail; render only ever reads.

Enums§

RunOutcome
Terminal disposition of one supervised run.

Constants§

METRICS_SCHEMA
Frozen metrics-schema version (RFC 0016 §4.1 / §8.1). Surfaced in the manifest at surfaces.metrics_schema; the integrator wires the surface — this const is the single source of truth this chunk owns. Additive series/label-values bump the minor; a removed/renamed metric or label key bumps the major (§8.2).

Functions§

record_config_reload
A config hot reload reached a terminal disposition (RFC 0017 §5.6). Drives agent_config_reload_total{result} with the closed domain applied|rejected (an unknown value buckets other). A rejected reload is a clean no-op (the running config is unchanged); applied bumps the generation gauge via set_config_generation.
record_drain
A drain phase transition (RFC 0011 §4 / RFC 0016 §4.3 agent_drains_total). phasestarted|completed|forced (an unknown value buckets other).
record_intel_call
An intelligence call was made (intel.call, RFC 0016 §4.3 agent_intel_calls_total).
record_intel_error
An intelligence-endpoint error by reason (RFC 0016 §4.3 agent_intel_errors_total). reasonunreachable|auth|timeout| 5xx (an unknown value buckets other).
record_limit_exceeded
A hard bound trip (limit.exceeded, RFC 0010 §3.3 / RFC 0016 §4.3).
record_loop_step
One loop step executed (loop.step, RFC 0010 §3.3). Drives agent_loop_steps_total (RFC 0016 §4.3).
record_mcp_connect_failure
An MCP connect attempt failed for a declared server (mcp.connect.fail, RFC 0016 §4.3 agent_mcp_connect_failures_total). server is the declared server name (bounded — there is a fixed, small declared set, RFC 0004); an over-capacity name buckets under other so the series stays bounded.
record_reaction
A reactive trigger fired (one reaction).
record_reactor_stall
A wedged-reactor liveness trip (RFC 0003 / RFC 0016 §5 / §4.3 agent_reactor_stalls_total).
record_refusal
A refusal / guard trip by reason (RFC 0016 §4.3 agent_refusals_total).
record_restart_tripped
The restart governor’s circuit breaker tripped.
record_run
A supervised run reached a terminal disposition.
record_run_started
A supervised run began (supervise_once entry).
record_run_status
A supervised run reached a terminal status — the frozen §4.3 form.
record_step
A workflow step reached a terminal status (agent_steps_total{status}).
record_store_op
A remote-store op completed (agent_store_ops_total{result} + latency sum).
record_subagent_exited
A subagent exited with a terminal status (subagent.exit, RFC 0007 §3.4). Drives agent_subagents_exited_total{status} (RFC 0016 §4.3).
record_subagent_restart
A subagent was restarted by the governor (subagent.restart, RFC 0003 §3.7). Drives agent_subagent_restarts_total{reason} (RFC 0016 §4.3).
record_subagent_spawned
A subagent was spawned (subagent.spawn, RFC 0016 §4.3).
record_subagent_stuck_kill
A wedged/stuck subagent was killed (subagent.stuck, RFC 0003 — the reliability headline). Drives agent_subagent_stuck_kills_total{signal} (RFC 0016 §4.3); signalterm|kill (an unknown value buckets other).
record_supervisor_restart
A supervisor process restart was observed (rebuild+reconcile, RFC 0003 §3.11). Drives agent_restarts_total (RFC 0016 §4.3) — distinct from the breaker-trip counter record_restart_tripped.
record_tokens
Tokens reported up by a direct child (AgentMsg::Usage).
record_turn
A turn worker ran (agent_turns_total{kind}, agentd 2.0).
set_budget_tokens_remaining
Point-in-time set of the lifetime-budget balance gauge (agent_budget_tokens_remaining, RFC 0025 §3.4): tokens left before the per-instance cumulative cap is reached — the alerting/scaling hook for the threshold event. Only ever set when a budget is installed (absent = the gauge stays at its 0 default, which a scraper reads together with the fact that no budget metric transitions occurred; unbounded instances simply never call this).
set_config_generation
Point-in-time set of the config-generation gauge (agent_config_generation, RFC 0017 §5.6): the count of successfully-applied reloads, so a scraper can detect “this instance has picked up generation N” against agentctl’s desired generation. Monotonic in practice (the reload loop only ever increments it).
set_context_tokens
Point-in-time set of the largest live context’s token estimate (agent_context_tokens).
set_inbox_pending
Point-in-time set of the durable inbox backlog (agent_inbox_pending).
set_intel_all_down
Point-in-time set of the intelligence all-endpoints-down gauge (agent_intel_all_down, RFC 0018 §6 / RFC 0016 §4.3) — 1 while every model endpoint is down (the latched, eventually-consistent last-child-experience truth a subagent reports up via AgentMsg::IntelHealth; the same flag flips /readyz NotReady). 0 once any endpoint is usable again. Distinct from agent_intel_up (the active endpoint’s reachability): all-down is the fleet-routing signal (no endpoint usable at all). No-op-safe / metrics-gated.
set_intel_up
Point-in-time set of the intelligence-endpoint reachability gauge (agent_intel_up, RFC 0016 §4.3 — RFC 0006/0018).
set_paused
Point-in-time set of the tree-pause gauge (agent_paused, RFC 0015 §5.5 / RFC 0016 §4.3) — 1 while the pause operator tool has frozen the agentic loops, 0 after resume. No-op-safe / metrics-gated, mirroring set_intel_up.
set_pressure
Point-in-time set of the resource-pressure gauges (agent_pressure_level, agent_disk_free_bytes): the shed/drain state the admission gates act on and the disk headroom that (usually) drives it. disk_free: None = no file store on this instance — the byte gauge is then not emitted at all, because exporting the supervisor’s local free space when durability lives elsewhere would invite alerts on the wrong disk.
set_reactive_backlog
Point-in-time set of the reactive backlog gauges — the RFC 0019 scaling signal set (RFC 0016 §4.3: agent_pending_events / agent_inflight_reactions / agent_subscriptions_active / agent_reaction_lag_ms).
set_tree_shape
Point-in-time set of the subagent-tree shape gauges (RFC 0016 §4.3: agent_active_subagents / agent_tree_depth / agent_tree_breadth).
set_work_backlog
Point-in-time set of the work-in-progress gauges (agent_runs_active, agent_turns_queued): non-terminal workflow runs, and conversation turns waiting for a dispatch slot (parallelism, pause, drain, or shed — the gauge does not say which; the event stream does).