Skip to main content

ExperimentMonitor

Struct ExperimentMonitor 

Source
pub struct ExperimentMonitor { /* private fields */ }
Expand description

Per-(tenant, route) certified-selection state. Cheap to share behind an Arc.

Implementations§

Source§

impl ExperimentMonitor

Source

pub fn from_config(config: &Config) -> Self

A monitor for every route in config that has a challenger. Routes without one are ignored by every method, so a deployment with no experiments pays nothing.

Source

pub fn unavailable(config: &Config) -> Self

A monitor that refuses to serve any challenger. Used when the history could not be read.

Source

pub fn with_log(self, log: Arc<dyn ExperimentLogBackend>) -> Self

Attach a shared log backend. Every observe_outcome/observe_pair call will append to the backend, and sync_from_backend will replay newly-arrived records from it. Returns self for method-chaining.

Source

pub fn with_paired_ledger(self, ledger: Arc<ShadowLedger>) -> Self

Attach a paired-challenger spend ledger. When set, apply_log_record seeds cluster spend into it so may_spend reflects all replicas’ spend, not just this process’s. Returns self for method-chaining. Only effective when a log backend is also attached.

Source

pub fn has_log(&self) -> bool

Whether a shared log backend is currently attached.

Source

pub async fn ensure_synced( &self, tenant: &str, route_ix: usize, fingerprint: &str, ) -> bool

Perform one sync_from_backend the first time a (tenant, route) entry is used in this process. Returns true when the entry is synced (either it already was, or the sync just succeeded). Returns false when the log backend is unreachable; callers should fail closed for this request (serve incumbent / skip evaluation).

Why this is needed. build_paired_ledger starts empty when shared_log=true, and start_sync_task only syncs entries already in the map. A brand-new replica (or a (tenant, route) pair not yet seen on this instance) would check challenger_blocked and may_spend against nothing, silently bypassing a cluster-level halt decision or the daily spend cap. One synchronous sync before the first check closes that window.

Burst coalescing. A per-entry tokio::sync::Mutex gate serialises concurrent first-request callers so exactly one LRANGE is issued per burst. Subsequent waiters re-check last_synced_at after acquiring the gate and return immediately. last_synced_at is set only on success; on failure it stays false (so the next burst retries), but a warning is logged only once per entry to avoid spam during a persistent outage.

Only the first request per (tenant, route_ix) per process pays one backend LRANGE.

Source

pub fn fingerprint(&self, route_ix: usize) -> Option<&str>

The fingerprint receipts for route_ix must carry to count, if it has a challenger.

Source

pub fn challenger_blocked(&self, tenant: &str, route_ix: usize) -> bool

Whether the serving path must NOT use the challenger for (tenant, route_ix): the experiment is halted, or its history was unreadable at startup.

Source

pub fn is_active(&self) -> bool

Whether any route runs an experiment.

Source

pub fn status(&self, tenant: &str, route_ix: usize) -> ExperimentStatus

Where the experiment for (tenant, route_ix) stands. ExperimentStatus::Undecided when nothing has been observed or the route has no challenger. The serving path stops using the challenger when this is ExperimentStatus::Halted.

Source

pub fn observe_cost( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, usd: f64, )

Record the cost of one served request of arm. Call with trace.final_.total_cost_usd.

fingerprint is the receipt’s; a mismatch with the current config counts the receipt in ignored_stale and records nothing.

Source

pub fn observe_health( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, errored: bool, truncated: u64, )

Tally one request of arm that errored and/or produced answers cut off at the output-token limit (truncated counts attempts). Stale fingerprints are ignored: observe_cost already counts the receipt once.

Source

pub fn observe_outcome( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, session_id: &str, verdict: Verdict, ) -> bool

Record a session’s task outcome. Returns whether it was counted.

Only the first clear outcome per session counts; later ones are ignored. The test pairs the k-th incumbent outcome with the k-th challenger outcome and treats sessions as independent draws, so several correlated outcomes from one long agent run would both overweight that session and break the independence the error bound rests on.

Pass is 1.0, Fail is 0.0, Abstain carries no information and is ignored.

Source

pub fn reports(&self) -> Vec<Snapshot>

A snapshot of every experiment, ordered by (tenant, route_ix).

Source

pub async fn sync_from_backend( &self, tenant: &str, route_ix: usize, fingerprint: &str, ) -> Result<(), LogError>

Read records appended to the shared log since the last sync and apply them locally.

Each call is cheap when there are no new records: one LRANGE that returns empty. Call this after every feedback event (within the feedback handler, not on the hot serve path) and from a background timer so all replicas stay ≤ 5 s stale.

Concurrent-call safety. record_pair and start_sync_task both call this concurrently on the same (tenant, route_ix). Both may snapshot the same remote_log_offset before either updates it, load the same slice from the backend, and then race to apply it. The fix is two-part:

  1. After re-acquiring the lock, skip any record whose absolute index offset + i is already covered by the current remote_log_offset (a concurrent call advanced it while we were awaiting load_since).
  2. Only ever advance remote_log_offset — never rewind it — so a slower concurrent call that loaded a shorter slice cannot wind the cursor back.

No-op (returns Ok) when no log backend is configured. Returns Err if the backend is unreachable; callers should treat a failure as “state unknown” and fail closed (do not serve the challenger for this request).

Source

pub async fn record_outcome( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, session_id: &str, verdict: Verdict, cost_usd: f64, ) -> bool

Record an outcome to the shared log (if configured), then sync from the log.

In single-replica mode (no log), falls back to the synchronous Self::observe_outcome. cost_usd is stored in the log so other replicas can tally costs without needing the full trace. Returns true when the outcome was counted.

Source

pub fn apply_pair_locally( &self, tenant: &str, route_ix: usize, fingerprint: &str, x_inc: f64, x_ch: f64, challenger_cost_usd: f64, trace_id: Uuid, ) -> bool

Apply a paired shadow result directly to local in-memory state.

Call only after the pair has been durably recorded (written to SQLite as NotInLog). This ordering guarantee is what spawn_paired_challenger enforces: the SQLite row is the only durable record for local-only pairs, and the evidence must not exist in memory without it — otherwise a crash between the evidence apply and the write lets a halted challenger run again on restart (it has evidence but no persisted state to replay the halt from).

Deduplicates by trace_id so a pair is never counted twice even if this method is called from both the local path and the log-sync path. Uuid::nil() bypasses dedup.

Returns true if the pair was accepted by the underlying Experiment.

Source

pub async fn record_pair( &self, tenant: &str, route_ix: usize, fingerprint: &str, x_inc: f64, x_ch: f64, incumbent_cost_usd: f64, challenger_cost_usd: f64, ch_error: bool, ch_truncated: u64, ts_unix: i64, trace_id: Uuid, incumbent_trace_id: Uuid, ) -> bool

Try to append a paired shadow result to the shared log (if configured), then sync.

Durability contract for spawn_paired_challenger:

  • Returns true → the record is in the shared log (the durable record). Evidence is applied via sync_from_backend inside this call. Caller writes the SQLite row as InLog after this returns (the row is just a startup-rebuild cache; it can follow the log write safely).
  • Returns false → NOT in the log (no log configured, or the append failed). Evidence is NOT applied here. The caller must write the SQLite row as NotInLog first, then call Self::apply_pair_locally only on success. This ordering ensures the durable record exists before evidence is in memory: a crash between the two would leave no evidence, which is the safe direction (a missed pair only slows the experiment; evidence without a durable record could mislead a halt).

ch_error, ch_truncated, and ts_unix are forwarded into LogRecord::Pair so other replicas can rebuild challenger health and seed the paired-spend ledger.

trace_id is used for cross-source dedup: if Redis applied the record but the append returned Err (timeout), the next sync sees the same trace_id and skips evidence it already counted locally. Pass Uuid::nil() in tests not exercising dedup.

Source

pub fn start_sync_task(self: &Arc<Self>, interval: Duration)

Spawn a background task that calls sync_from_backend for every known experiment at most every interval (spec: ≤ 5 s). The task exits silently when the Arc becomes the last reference. No-op when no log backend is configured.

Trait Implementations§

Source§

impl Debug for ExperimentMonitor

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for ExperimentMonitor

Source§

fn default() -> Self

An inert monitor with no experiments configured: every lookup returns false/unknown, no log is attached, and no ledger is attached. Used in demo/test paths that construct an AppState without a real Config (e.g. Arc::default()).

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T> Instrument for T

Source§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided Span, returning an Instrumented wrapper. Read more
Source§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

impl<Unshared, Shared> IntoShared<Shared> for Unshared
where Shared: FromUnshared<Unshared>,

Source§

fn into_shared(self) -> Shared

Creates a shared type from an unshared type.
Source§

impl<T> Pointable for T

Source§

const ALIGN: usize

The alignment of pointer.
Source§

type Init = T

The type for initializers.
Source§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
Source§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
Source§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
Source§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
Source§

impl<T> PolicyExt for T
where T: ?Sized,

Source§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow only if self and other return Action::Follow. Read more
Source§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns Action::Follow if either self or other returns Action::Follow. Read more
Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V

Source§

impl<T> WithSubscriber for T

Source§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a WithDispatch wrapper. Read more
Source§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a WithDispatch wrapper. Read more