Module metrics 

Process-local counters → Prometheus text. [feature: metrics]

Off the default path: the public record_* fns are no-ops unless built with --features metrics, so call sites stay clean and the default build pays nothing (metrics are otherwise derivable from the JSON-lines event stream — that is the default story). With the feature, a tiny dependency-free atomic registry backs an opt-in HTTP /metrics scrape surface (obs::serve).

Counters are per supervisor process. The long-lived root daemon’s surface reflects the runs it supervises — every one-shot, reaction, and scheduled fire flows through supervise_once — plus the tokens its direct children report up the control channel. Nested subagents keep their own (process-local) counters, still visible in their logs; cross-process metric rollup is a deliberate non-goal (the same process boundary the tree token ceiling draws).

The frozen metrics_schema contract

The metric names and label keys below are a versioned public API (METRICS_SCHEMA) that a control plane (agentctl) authors dashboards, alerts and scalers against. Exposition is hand-written Prometheus 0.0.4 text — no prometheus/metrics crate. The enumerated set is the contract: it is additive within a major, and removing or renaming a metric or a label key bumps the major.

Cardinality is binding: /metrics is unauthenticated and may be bound on all interfaces. Labels carry bounded values only (status, model, type, server, tool, reason, limit, signal, phase, ok); never run_id / agent_id / agent_path / call_id / a resource URI — those are unbounded and live in logs and traces only. A control plane that needs per-run granularity reads the run report or the event stream, never a metric. This module therefore stores label-bearing series as small fixed-domain atomic arrays (the closed label set is known at compile time), so the cardinality is structurally bounded.

Telemetry never crashes the agent: every fn here is a plain atomic add/store that cannot fail; render only ever reads.

Enums

RunOutcome

Terminal disposition of one supervised run.

Constants

METRICS_SCHEMA

Frozen metrics-schema version. Surfaced in the manifest at surfaces.metrics_schema; the integrator wires that surface — this const is the single source of truth for the value. Additive series and label values bump the minor; a removed or renamed metric or label key bumps the major.

Functions

record_config_reload

A config hot reload reached a terminal disposition. Drives agent_config_reload_total{result} with the closed domain applied|rejected (an unknown value buckets other). A rejected reload is a clean no-op (the running config is unchanged); applied bumps the generation gauge via set_config_generation.

record_drain

A drain phase transition. Drives agent_drains_total. phase ∈ started|completed|forced (an unknown value buckets other).

record_intel_call

An intelligence call was made (intel.call). Drives agent_intel_calls_total.

record_intel_error

An intelligence-endpoint error by reason. Drives agent_intel_errors_total. reason ∈ unreachable|auth|timeout| 5xx (an unknown value buckets other).

record_limit_exceeded

A hard bound trip (limit.exceeded).

record_loop_step

One loop step executed (loop.step). Drives agent_loop_steps_total.

record_mcp_connect_failure

An MCP connect attempt failed for a declared server (mcp.connect.fail). Drives agent_mcp_connect_failures_total. server is the declared server name — bounded, because the declared set is small and fixed at config time; an over-capacity name buckets under other so the series stays bounded.

record_reaction

A reactive trigger fired (one reaction).

record_reactor_stall

A wedged-reactor liveness trip. Drives agent_reactor_stalls_total.

record_refusal

A refusal / guard trip by reason (drives agent_refusals_total).

record_restart_tripped

The restart governor’s circuit breaker tripped.

record_run

A supervised run reached a terminal disposition.

record_run_started

A supervised run began (supervise_once entry).

record_run_status

A supervised run reached a terminal status — the frozen, precise form.

record_step

A workflow step reached a terminal status (agent_steps_total{status}).

record_store_op

A remote-store op completed (agent_store_ops_total{result} + latency sum).

record_subagent_exited

A subagent exited with a terminal status (subagent.exit). Drives agent_subagents_exited_total{status} over the closed status vocabulary.

record_subagent_restart

A subagent was restarted by the governor (subagent.restart). Drives agent_subagent_restarts_total{reason}.

record_subagent_spawned

A subagent was spawned (subagent.spawn).

record_subagent_stuck_kill

A wedged/stuck subagent was killed (subagent.stuck — the reliability headline). Drives agent_subagent_stuck_kills_total{signal}; signal ∈ term|kill (an unknown value buckets other).

record_supervisor_restart

A supervisor process restart was observed (rebuild + reconcile). Drives agent_restarts_total — distinct from the breaker-trip counter record_restart_tripped.

record_tokens

Tokens reported up by a direct child (AgentMsg::Usage).

record_turn

A turn worker ran (agent_turns_total{kind}, agentd).

set_budget_tokens_remaining

Point-in-time set of the lifetime-budget balance gauge (agent_budget_tokens_remaining): tokens left before the per-instance cumulative cap is reached — the alerting/scaling hook for the threshold event. Only ever set when a budget is installed (absent = the gauge stays at its 0 default, which a scraper reads together with the fact that no budget metric transitions occurred; unbounded instances simply never call this).

set_config_generation

Point-in-time set of the config-generation gauge (agent_config_generation): the count of successfully-applied reloads, so a scraper can detect “this instance has picked up generation N” against agentctl’s desired generation. Monotonic in practice — the reload loop only ever increments it.

set_context_tokens

Point-in-time set of the largest live context’s token estimate (agent_context_tokens).

set_inbox_pending

Point-in-time set of the durable inbox backlog (agent_inbox_pending).

set_intel_all_down

Point-in-time set of the intelligence all-endpoints-down gauge (agent_intel_all_down) — 1 while every model endpoint is down (the latched, eventually-consistent last-child-experience truth a subagent reports up via AgentMsg::IntelHealth; the same flag flips /readyz NotReady). 0 once any endpoint is usable again. Distinct from agent_intel_up (the active endpoint’s reachability): all-down is the fleet-routing signal (no endpoint usable at all). No-op-safe / metrics-gated.

set_intel_up

Point-in-time set of the intelligence-endpoint reachability gauge (agent_intel_up).

set_paused

Point-in-time set of the tree-pause gauge (agent_paused) — 1 while the pause operator tool has frozen the agentic loops, 0 after resume. No-op-safe / metrics-gated, mirroring set_intel_up.

set_pressure

Point-in-time set of the resource-pressure gauges (agent_pressure_level, agent_disk_free_bytes): the shed/drain state the admission gates act on and the disk headroom that (usually) drives it. disk_free: None = no file store on this instance — the byte gauge is then not emitted at all, because exporting the supervisor’s local free space when durability lives elsewhere would invite alerts on the wrong disk.

set_reactive_backlog

Point-in-time set of the reactive backlog gauges — the scaling signal set an autoscaler reads (agent_pending_events / agent_inflight_reactions / agent_subscriptions_active / agent_reaction_lag_ms).

set_tree_shape

Point-in-time set of the subagent-tree shape gauges (agent_active_subagents / agent_tree_depth / agent_tree_breadth).

set_work_backlog

Point-in-time set of the work-in-progress gauges (agent_runs_active, agent_turns_queued): non-terminal workflow runs, and conversation turns waiting for a dispatch slot (parallelism, pause, drain, or shed — the gauge does not say which; the event stream does).