Expand description
Live monitor for certified model selection (ADR 0014).
A route with a [route.challenger] serves a randomized fraction of sessions from a second
ladder. This module owns the evidence: one Experiment per (tenant, route), fed with the
cost of every served experiment request and the first task outcome of every session. Its two
jobs are the halt guard (the serving path asks ExperimentMonitor::status and stops using a
challenger that has been proven worse) and the report an operator reads before adopting one.
§What counts as an outcome
Feedback with a pass or fail verdict counts, whatever its gate_id; abstain is
ignored, and only the first counted outcome per session is used. Nothing here can tell a CI
result from a human rating, so operators should post exactly one thing per session: the task
outcome the experiment is meant to compare. Posting several different signals under different
gate ids makes “first one wins” arbitrary.
§Configuration identity
Every receipt carries a fingerprint of the experiment configuration (both ladders, margin,
alpha). Receipts whose fingerprint differs from the current config’s are ignored and counted
(ignored_stale), so changing either ladder or the margin/alpha starts a fresh experiment
instead of pooling evidence about a different challenger.
Tenants never share an experiment (ADR 0004): the key is (tenant, route_ix) and nothing here
reads across tenants.
The same state is re-derivable from receipts. experiments_from_traces replays the stored
traces and deferred verdicts through the very same monitor, so a restart (or the
firstpass experiment CLI) lands on the decision the live process would have reached.
§Shared state across replicas (opt-in)
When ExperimentMonitor is built with a ExperimentLogBackend (set via
ExperimentMonitor::with_log), every outcome (live mode) or pair (paired mode) is appended
to a shared log keyed by (tenant, route_ix, fingerprint). Each replica re-reads the log on
every feedback event and on a periodic timer (≤ 5 s), rebuilding its local Experiment from
the canonical arrival order. Because the e-process is deterministic, all replicas holding the
same log reach the same halt/certify decision. Without a backend the behaviour is unchanged.
Structs§
- ArmHealth
- Requests an arm could not answer cleanly. Neither kind is random: errors (timeouts above all)
and output-limit cut-offs cluster on long, hard requests. An incumbent error drops its pair, so
many of them mean the pass rates describe an easier workload than the route serves; a
challenger error counts as a failed pair. Truncated answers are graded as they are, so a high
count says the route’s
max_tokensmay be deciding the comparison. - Experiment
Monitor - Per-
(tenant, route)certified-selection state. Cheap to share behind anArc. - Experiment
View - One experiment as the CLI presents it.
- Health
ArmHealthfor both arms. For a paired challenger the counts are persisted inexperiment_pairsand inLogRecord::Pair, so they survive restarts and are shared across replicas when a shared log backend is attached.- InMemory
Experiment Log - In-process log for tests (and the no-Redis single-replica path, where it is never used).
- Snapshot
- One experiment as the monitor holds it.
Enums§
- LogError
- An error returned by
ExperimentLogBackendoperations. - LogRecord
- A record appended to the shared experiment log.
- Recommendation
- What an operator should do with an experiment’s current state.
Traits§
- Experiment
LogBackend - A shared log that lets several replicas converge on the same
Experimentstate.
Functions§
- build_
monitor - The monitor to serve with, rebuilt from the trace store at startup.
- build_
monitor_ with build_monitor, told whether a shared experiment log will be attached.- experiments_
from_ traces - Replay receipts into per-
(tenant, route)snapshots. Seemonitor_from_traces. - is_
incumbent_ trace - Whether a trace may be used for rung-indexed or threshold analyses (
ope,calibrate). Challenger-arm traces are excluded: their rungs index a different ladder and their gate scores relate to correctness differently, so pooling them would silently mix two models’ behaviour. - monitor_
from_ traces - Rebuild a monitor from receipts.
- recommend
- The recommendation for a report. Cost is compared as dollars per served request on each arm; missing cost data on either arm counts as “no saving” because adoption must not rest on a number that was never measured.
- render
- Human-readable report, one block per
(tenant, route). - truncated_
attempts - Number of attempts in
tthat stopped at the output-token limit. - views
- Pair each report with its recommendation and the challenger ladder from
config.