Skip to main content

Module experiment

Module experiment 

Source
Expand description

Live monitor for certified model selection (ADR 0014).

A route with a [route.challenger] serves a randomized fraction of sessions from a second ladder. This module owns the evidence: one Experiment per (tenant, route), fed with the cost of every served experiment request and the first task outcome of every session. Its two jobs are the halt guard (the serving path asks ExperimentMonitor::status and stops using a challenger that has been proven worse) and the report an operator reads before adopting one.

§What counts as an outcome

Feedback with a pass or fail verdict counts, whatever its gate_id; abstain is ignored, and only the first counted outcome per session is used. Nothing here can tell a CI result from a human rating, so operators should post exactly one thing per session: the task outcome the experiment is meant to compare. Posting several different signals under different gate ids makes “first one wins” arbitrary.

§Configuration identity

Every receipt carries a fingerprint of the experiment configuration (both ladders, margin, alpha). Receipts whose fingerprint differs from the current config’s are ignored and counted (ignored_stale), so changing either ladder or the margin/alpha starts a fresh experiment instead of pooling evidence about a different challenger.

Tenants never share an experiment (ADR 0004): the key is (tenant, route_ix) and nothing here reads across tenants.

The same state is re-derivable from receipts. experiments_from_traces replays the stored traces and deferred verdicts through the very same monitor, so a restart (or the firstpass experiment CLI) lands on the decision the live process would have reached.

§Shared state across replicas (opt-in)

When ExperimentMonitor is built with a ExperimentLogBackend (set via ExperimentMonitor::with_log), every outcome (live mode) or pair (paired mode) is appended to a shared log keyed by (tenant, route_ix, fingerprint). Each replica re-reads the log on every feedback event and on a periodic timer (≤ 5 s), rebuilding its local Experiment from the canonical arrival order. Because the e-process is deterministic, all replicas holding the same log reach the same halt/certify decision. Without a backend the behaviour is unchanged.

Structs§

ArmHealth
Requests an arm could not answer cleanly. Neither kind is random: errors (timeouts above all) and output-limit cut-offs cluster on long, hard requests. An incumbent error drops its pair, so many of them mean the pass rates describe an easier workload than the route serves; a challenger error counts as a failed pair. Truncated answers are graded as they are, so a high count says the route’s max_tokens may be deciding the comparison.
ExperimentMonitor
Per-(tenant, route) certified-selection state. Cheap to share behind an Arc.
ExperimentView
One experiment as the CLI presents it.
Health
ArmHealth for both arms. For a paired challenger the counts are persisted in experiment_pairs and in LogRecord::Pair, so they survive restarts and are shared across replicas when a shared log backend is attached.
InMemoryExperimentLog
In-process log for tests (and the no-Redis single-replica path, where it is never used).
Snapshot
One experiment as the monitor holds it.

Enums§

LogError
An error returned by ExperimentLogBackend operations.
LogRecord
A record appended to the shared experiment log.
Recommendation
What an operator should do with an experiment’s current state.

Traits§

ExperimentLogBackend
A shared log that lets several replicas converge on the same Experiment state.

Functions§

build_monitor
The monitor to serve with, rebuilt from the trace store at startup.
build_monitor_with
build_monitor, told whether a shared experiment log will be attached.
experiments_from_traces
Replay receipts into per-(tenant, route) snapshots. See monitor_from_traces.
is_incumbent_trace
Whether a trace may be used for rung-indexed or threshold analyses (ope, calibrate). Challenger-arm traces are excluded: their rungs index a different ladder and their gate scores relate to correctness differently, so pooling them would silently mix two models’ behaviour.
monitor_from_traces
Rebuild a monitor from receipts.
recommend
The recommendation for a report. Cost is compared as dollars per served request on each arm; missing cost data on either arm counts as “no saving” because adoption must not rest on a number that was never measured.
render
Human-readable report, one block per (tenant, route).
truncated_attempts
Number of attempts in t that stopped at the output-token limit.
views
Pair each report with its recommendation and the challenger ladder from config.