pub struct ExperimentMonitor { /* private fields */ }Expand description
Per-(tenant, route) certified-selection state. Cheap to share behind an Arc.
Implementations§
Source§impl ExperimentMonitor
impl ExperimentMonitor
Sourcepub fn from_config(config: &Config) -> Self
pub fn from_config(config: &Config) -> Self
A monitor for every route in config that has a challenger. Routes without one are ignored
by every method, so a deployment with no experiments pays nothing.
A monitor that refuses to serve any challenger. Used when the history could not be read.
Sourcepub fn with_log(self, log: Arc<dyn ExperimentLogBackend>) -> Self
pub fn with_log(self, log: Arc<dyn ExperimentLogBackend>) -> Self
Attach a shared log backend. Every observe_outcome/observe_pair call will append to
the backend, and sync_from_backend will replay newly-arrived records from it. Returns
self for method-chaining.
Sourcepub fn with_paired_ledger(self, ledger: Arc<ShadowLedger>) -> Self
pub fn with_paired_ledger(self, ledger: Arc<ShadowLedger>) -> Self
Attach a paired-challenger spend ledger. When set, apply_log_record seeds cluster spend
into it so may_spend reflects all replicas’ spend, not just this process’s. Returns
self for method-chaining. Only effective when a log backend is also attached.
Sourcepub async fn ensure_synced(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
) -> bool
pub async fn ensure_synced( &self, tenant: &str, route_ix: usize, fingerprint: &str, ) -> bool
Perform one sync_from_backend the first time a (tenant, route) entry is used in this
process. Returns true when the entry is synced (either it already was, or the sync just
succeeded). Returns false when the log backend is unreachable; callers should fail closed
for this request (serve incumbent / skip evaluation).
Why this is needed. build_paired_ledger starts empty when shared_log=true, and
start_sync_task only syncs entries already in the map. A brand-new replica (or a
(tenant, route) pair not yet seen on this instance) would check challenger_blocked and
may_spend against nothing, silently bypassing a cluster-level halt decision or the daily
spend cap. One synchronous sync before the first check closes that window.
Burst coalescing. A per-entry tokio::sync::Mutex gate serialises concurrent
first-request callers so exactly one LRANGE is issued per burst. Subsequent waiters
re-check last_synced_at after acquiring the gate and return immediately. last_synced_at is
set only on success; on failure it stays false (so the next burst retries), but a warning
is logged only once per entry to avoid spam during a persistent outage.
Only the first request per (tenant, route_ix) per process pays one backend LRANGE.
Sourcepub fn fingerprint(&self, route_ix: usize) -> Option<&str>
pub fn fingerprint(&self, route_ix: usize) -> Option<&str>
The fingerprint receipts for route_ix must carry to count, if it has a challenger.
Sourcepub fn challenger_blocked(&self, tenant: &str, route_ix: usize) -> bool
pub fn challenger_blocked(&self, tenant: &str, route_ix: usize) -> bool
Whether the serving path must NOT use the challenger for (tenant, route_ix): the
experiment is halted, or its history was unreadable at startup.
Sourcepub fn status(&self, tenant: &str, route_ix: usize) -> ExperimentStatus
pub fn status(&self, tenant: &str, route_ix: usize) -> ExperimentStatus
Where the experiment for (tenant, route_ix) stands. ExperimentStatus::Undecided when
nothing has been observed or the route has no challenger. The serving path stops using the
challenger when this is ExperimentStatus::Halted.
Sourcepub fn observe_cost(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
usd: f64,
)
pub fn observe_cost( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, usd: f64, )
Record the cost of one served request of arm. Call with trace.final_.total_cost_usd.
fingerprint is the receipt’s; a mismatch with the current config counts the receipt in
ignored_stale and records nothing.
Sourcepub fn observe_health(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
errored: bool,
truncated: u64,
)
pub fn observe_health( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, errored: bool, truncated: u64, )
Tally one request of arm that errored and/or produced answers cut off at the output-token
limit (truncated counts attempts). Stale fingerprints are ignored: observe_cost already
counts the receipt once.
Sourcepub fn observe_outcome(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
session_id: &str,
verdict: Verdict,
) -> bool
pub fn observe_outcome( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, session_id: &str, verdict: Verdict, ) -> bool
Record a session’s task outcome. Returns whether it was counted.
Only the first clear outcome per session counts; later ones are ignored. The test pairs the k-th incumbent outcome with the k-th challenger outcome and treats sessions as independent draws, so several correlated outcomes from one long agent run would both overweight that session and break the independence the error bound rests on.
Pass is 1.0, Fail is 0.0, Abstain carries no information and is ignored.
Sourcepub fn reports(&self) -> Vec<Snapshot>
pub fn reports(&self) -> Vec<Snapshot>
A snapshot of every experiment, ordered by (tenant, route_ix).
Sourcepub async fn sync_from_backend(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
) -> Result<(), LogError>
pub async fn sync_from_backend( &self, tenant: &str, route_ix: usize, fingerprint: &str, ) -> Result<(), LogError>
Read records appended to the shared log since the last sync and apply them locally.
Each call is cheap when there are no new records: one LRANGE that returns empty. Call this after every feedback event (within the feedback handler, not on the hot serve path) and from a background timer so all replicas stay ≤ 5 s stale.
Concurrent-call safety. record_pair and start_sync_task both call this
concurrently on the same (tenant, route_ix). Both may snapshot the same
remote_log_offset before either updates it, load the same slice from the backend,
and then race to apply it. The fix is two-part:
- After re-acquiring the lock, skip any record whose absolute index
offset + iis already covered by the currentremote_log_offset(a concurrent call advanced it while we were awaitingload_since). - Only ever advance
remote_log_offset— never rewind it — so a slower concurrent call that loaded a shorter slice cannot wind the cursor back.
No-op (returns Ok) when no log backend is configured.
Returns Err if the backend is unreachable; callers should treat a failure as
“state unknown” and fail closed (do not serve the challenger for this request).
Sourcepub async fn record_outcome(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
session_id: &str,
verdict: Verdict,
cost_usd: f64,
) -> bool
pub async fn record_outcome( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, session_id: &str, verdict: Verdict, cost_usd: f64, ) -> bool
Record an outcome to the shared log (if configured), then sync from the log.
In single-replica mode (no log), falls back to the synchronous Self::observe_outcome.
cost_usd is stored in the log so other replicas can tally costs without needing the
full trace. Returns true when the outcome was counted.
Sourcepub fn apply_pair_locally(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
x_inc: f64,
x_ch: f64,
challenger_cost_usd: f64,
trace_id: Uuid,
) -> bool
pub fn apply_pair_locally( &self, tenant: &str, route_ix: usize, fingerprint: &str, x_inc: f64, x_ch: f64, challenger_cost_usd: f64, trace_id: Uuid, ) -> bool
Apply a paired shadow result directly to local in-memory state.
Call only after the pair has been durably recorded (written to SQLite as NotInLog).
This ordering guarantee is what spawn_paired_challenger enforces: the SQLite row is the
only durable record for local-only pairs, and the evidence must not exist in memory without
it — otherwise a crash between the evidence apply and the write lets a halted challenger
run again on restart (it has evidence but no persisted state to replay the halt from).
Deduplicates by trace_id so a pair is never counted twice even if this method is called
from both the local path and the log-sync path. Uuid::nil() bypasses dedup.
Returns true if the pair was accepted by the underlying Experiment.
Sourcepub async fn record_pair(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
x_inc: f64,
x_ch: f64,
incumbent_cost_usd: f64,
challenger_cost_usd: f64,
ch_error: bool,
ch_truncated: u64,
ts_unix: i64,
trace_id: Uuid,
incumbent_trace_id: Uuid,
) -> bool
pub async fn record_pair( &self, tenant: &str, route_ix: usize, fingerprint: &str, x_inc: f64, x_ch: f64, incumbent_cost_usd: f64, challenger_cost_usd: f64, ch_error: bool, ch_truncated: u64, ts_unix: i64, trace_id: Uuid, incumbent_trace_id: Uuid, ) -> bool
Try to append a paired shadow result to the shared log (if configured), then sync.
Durability contract for spawn_paired_challenger:
- Returns
true→ the record is in the shared log (the durable record). Evidence is applied viasync_from_backendinside this call. Caller writes the SQLite row asInLogafter this returns (the row is just a startup-rebuild cache; it can follow the log write safely). - Returns
false→ NOT in the log (no log configured, or the append failed). Evidence is NOT applied here. The caller must write the SQLite row asNotInLogfirst, then callSelf::apply_pair_locallyonly on success. This ordering ensures the durable record exists before evidence is in memory: a crash between the two would leave no evidence, which is the safe direction (a missed pair only slows the experiment; evidence without a durable record could mislead a halt).
ch_error, ch_truncated, and ts_unix are forwarded into LogRecord::Pair so other
replicas can rebuild challenger health and seed the paired-spend ledger.
trace_id is used for cross-source dedup: if Redis applied the record but the append
returned Err (timeout), the next sync sees the same trace_id and skips evidence it
already counted locally. Pass Uuid::nil() in tests not exercising dedup.
Sourcepub fn start_sync_task(self: &Arc<Self>, interval: Duration)
pub fn start_sync_task(self: &Arc<Self>, interval: Duration)
Spawn a background task that calls sync_from_backend for every known experiment at
most every interval (spec: ≤ 5 s). The task exits silently when the Arc becomes
the last reference. No-op when no log backend is configured.
Trait Implementations§
Source§impl Debug for ExperimentMonitor
impl Debug for ExperimentMonitor
Auto Trait Implementations§
impl !Freeze for ExperimentMonitor
impl !RefUnwindSafe for ExperimentMonitor
impl !UnwindSafe for ExperimentMonitor
impl Send for ExperimentMonitor
impl Sync for ExperimentMonitor
impl Unpin for ExperimentMonitor
impl UnsafeUnpin for ExperimentMonitor
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more