Expand description
Process-local counters → Prometheus text. RFC 0010 §metrics. [feature: metrics]
Off the default path: the public record_* fns are no-ops unless built
with --features metrics, so call sites stay clean and the default build
pays nothing (metrics are otherwise derivable from the JSON-lines event
stream — that is the default story). With the feature, a tiny dependency-free
atomic registry backs an opt-in HTTP /metrics scrape surface (obs::serve).
Counters are per supervisor process. The long-lived root daemon’s surface
reflects the runs it supervises — every one-shot, reaction, and scheduled fire
flows through supervise_once — plus the tokens its direct children report
up the control channel. Nested subagents keep their own (process-local)
counters, still visible in their logs; cross-process metric rollup is a
deliberate non-goal for v1 (same boundary as the tree token ceiling).
§The frozen metrics_schema contract (RFC 0016 §4)
RFC 0016 §4 freezes the metric names and label keys below into a
versioned public API (METRICS_SCHEMA) a control plane (agentctl) authors
dashboards/alerts/scalers against. The exposition mechanism is unchanged from
RFC 0010 §3.8 (hand-written Prometheus 0.0.4 text — no prometheus/metrics
crate). The §4.3 enumerated set is the contract; it is additive within the
major and a removal/rename bumps the major (§8.2).
Cardinality (RFC 0016 §4.2, binding): /metrics is unauthenticated and
may be bound on all interfaces. Labels carry bounded values only
(status, model, type, server, tool, reason, limit, signal,
phase, ok); never run_id / agent_id / agent_path / call_id /
a resource URI — those are unbounded and live in logs/traces only. A control
plane that needs per-run granularity reads the run report (§6) or the event
stream (§7), never a metric. This module therefore stores label-bearing
series as small fixed-domain atomic arrays (the closed label set is known
at compile time), so the cardinality is structurally bounded.
Telemetry never crashes the agent (RFC 0016 §8.4): every fn here is a plain
atomic add/store that cannot fail; render only ever reads.
Enums§
- RunOutcome
- Terminal disposition of one supervised run.
Constants§
- METRICS_
SCHEMA - Frozen metrics-schema version (RFC 0016 §4.1 / §8.1). Surfaced in the manifest
at
surfaces.metrics_schema; the integrator wires the surface — this const is the single source of truth this chunk owns. Additive series/label-values bump the minor; a removed/renamed metric or label key bumps the major (§8.2).
Functions§
- record_
config_ reload - A config hot reload reached a terminal disposition (RFC 0017 §5.6). Drives
agent_config_reload_total{result}with the closed domainapplied|rejected(an unknown value bucketsother). Arejectedreload is a clean no-op (the running config is unchanged);appliedbumps the generation gauge viaset_config_generation. - record_
drain - A drain phase transition (RFC 0011 §4 / RFC 0016 §4.3
agent_drains_total).phase∈started|completed|forced(an unknown value bucketsother). - record_
intel_ call - An intelligence call was made (
intel.call, RFC 0016 §4.3agent_intel_calls_total). - record_
intel_ error - An intelligence-endpoint error by reason (RFC 0016 §4.3
agent_intel_errors_total).reason∈unreachable|auth|timeout|5xx(an unknown value bucketsother). - record_
limit_ exceeded - A hard bound trip (
limit.exceeded, RFC 0010 §3.3 / RFC 0016 §4.3). - record_
loop_ step - One loop step executed (
loop.step, RFC 0010 §3.3). Drivesagent_loop_steps_total(RFC 0016 §4.3). - record_
mcp_ connect_ failure - An MCP connect attempt failed for a declared
server(mcp.connect.fail, RFC 0016 §4.3agent_mcp_connect_failures_total).serveris the declared server name (bounded — there is a fixed, small declared set, RFC 0004); an over-capacity name buckets underotherso the series stays bounded. - record_
reaction - A reactive trigger fired (one reaction).
- record_
reactor_ stall - A wedged-reactor liveness trip (RFC 0003 / RFC 0016 §5 / §4.3
agent_reactor_stalls_total). - record_
refusal - A refusal / guard trip by reason (RFC 0016 §4.3
agent_refusals_total). - record_
restart_ tripped - The restart governor’s circuit breaker tripped.
- record_
run - A supervised run reached a terminal disposition.
- record_
run_ started - A supervised run began (
supervise_onceentry). - record_
run_ status - A supervised run reached a terminal status — the frozen §4.3 form.
- record_
step - A workflow step reached a terminal status (
agent_steps_total{status}). - record_
store_ op - A remote-store op completed (
agent_store_ops_total{result}+ latency sum). - record_
subagent_ exited - A subagent exited with a terminal
status(subagent.exit, RFC 0007 §3.4). Drivesagent_subagents_exited_total{status}(RFC 0016 §4.3). - record_
subagent_ restart - A subagent was restarted by the governor (
subagent.restart, RFC 0003 §3.7). Drivesagent_subagent_restarts_total{reason}(RFC 0016 §4.3). - record_
subagent_ spawned - A subagent was spawned (
subagent.spawn, RFC 0016 §4.3). - record_
subagent_ stuck_ kill - A wedged/stuck subagent was killed (
subagent.stuck, RFC 0003 — the reliability headline). Drivesagent_subagent_stuck_kills_total{signal}(RFC 0016 §4.3);signal∈term|kill(an unknown value bucketsother). - record_
supervisor_ restart - A supervisor process restart was observed (rebuild+reconcile, RFC 0003 §3.11).
Drives
agent_restarts_total(RFC 0016 §4.3) — distinct from the breaker-trip counterrecord_restart_tripped. - record_
tokens - Tokens reported up by a direct child (
AgentMsg::Usage). - record_
turn - A turn worker ran (
agent_turns_total{kind}, agentd 2.0). - set_
budget_ tokens_ remaining - Point-in-time set of the lifetime-budget balance gauge
(
agent_budget_tokens_remaining, RFC 0025 §3.4): tokens left before the per-instance cumulative cap is reached — the alerting/scaling hook for the threshold event. Only ever set when a budget is installed (absent = the gauge stays at its 0 default, which a scraper reads together with the fact that no budget metric transitions occurred; unbounded instances simply never call this). - set_
config_ generation - Point-in-time set of the config-generation gauge (
agent_config_generation, RFC 0017 §5.6): the count of successfully-applied reloads, so a scraper can detect “this instance has picked up generation N” against agentctl’s desired generation. Monotonic in practice (the reload loop only ever increments it). - set_
context_ tokens - Point-in-time set of the largest live context’s token estimate
(
agent_context_tokens). - set_
inbox_ pending - Point-in-time set of the durable inbox backlog (
agent_inbox_pending). - set_
intel_ all_ down - Point-in-time set of the intelligence all-endpoints-down gauge
(
agent_intel_all_down, RFC 0018 §6 / RFC 0016 §4.3) — 1 while every model endpoint is down (the latched, eventually-consistent last-child-experience truth a subagent reports up viaAgentMsg::IntelHealth; the same flag flips/readyzNotReady). 0 once any endpoint is usable again. Distinct fromagent_intel_up(the active endpoint’s reachability): all-down is the fleet-routing signal (no endpoint usable at all). No-op-safe / metrics-gated. - set_
intel_ up - Point-in-time set of the intelligence-endpoint reachability gauge
(
agent_intel_up, RFC 0016 §4.3 — RFC 0006/0018). - set_
paused - Point-in-time set of the tree-pause gauge (
agent_paused, RFC 0015 §5.5 / RFC 0016 §4.3) — 1 while thepauseoperator tool has frozen the agentic loops, 0 afterresume. No-op-safe / metrics-gated, mirroringset_intel_up. - set_
pressure - Point-in-time set of the resource-pressure gauges (
agent_pressure_level,agent_disk_free_bytes): the shed/drain state the admission gates act on and the disk headroom that (usually) drives it.disk_free: None= no file store on this instance — the byte gauge is then not emitted at all, because exporting the supervisor’s local free space when durability lives elsewhere would invite alerts on the wrong disk. - set_
reactive_ backlog - Point-in-time set of the reactive backlog gauges — the RFC 0019 scaling signal
set (RFC 0016 §4.3:
agent_pending_events/agent_inflight_reactions/agent_subscriptions_active/agent_reaction_lag_ms). - set_
tree_ shape - Point-in-time set of the subagent-tree shape gauges (RFC 0016 §4.3:
agent_active_subagents/agent_tree_depth/agent_tree_breadth). - set_
work_ backlog - Point-in-time set of the work-in-progress gauges (
agent_runs_active,agent_turns_queued): non-terminal workflow runs, and conversation turns waiting for a dispatch slot (parallelism, pause, drain, or shed — the gauge does not say which; the event stream does).