pub struct ExperimentMonitor { /* private fields */ }Expand description
Per-(tenant, route) certified-selection state. Cheap to share behind an Arc.
Implementations§
Source§impl ExperimentMonitor
impl ExperimentMonitor
Sourcepub fn from_config(config: &Config) -> Self
pub fn from_config(config: &Config) -> Self
A monitor for every route in config that has a challenger. Routes without one are ignored
by every method, so a deployment with no experiments pays nothing.
A monitor that refuses to serve any challenger. Used when the history could not be read.
Sourcepub fn with_log(self, log: Arc<dyn ExperimentLogBackend>) -> Self
pub fn with_log(self, log: Arc<dyn ExperimentLogBackend>) -> Self
Attach a shared log backend. Every observe_outcome/observe_pair call will append to
the backend, and sync_from_backend will replay newly-arrived records from it. Returns
self for method-chaining.
Sourcepub fn fingerprint(&self, route_ix: usize) -> Option<&str>
pub fn fingerprint(&self, route_ix: usize) -> Option<&str>
The fingerprint receipts for route_ix must carry to count, if it has a challenger.
Sourcepub fn challenger_blocked(&self, tenant: &str, route_ix: usize) -> bool
pub fn challenger_blocked(&self, tenant: &str, route_ix: usize) -> bool
Whether the serving path must NOT use the challenger for (tenant, route_ix): the
experiment is halted, or its history was unreadable at startup.
Sourcepub fn status(&self, tenant: &str, route_ix: usize) -> ExperimentStatus
pub fn status(&self, tenant: &str, route_ix: usize) -> ExperimentStatus
Where the experiment for (tenant, route_ix) stands. ExperimentStatus::Undecided when
nothing has been observed or the route has no challenger. The serving path stops using the
challenger when this is ExperimentStatus::Halted.
Sourcepub fn observe_cost(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
usd: f64,
)
pub fn observe_cost( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, usd: f64, )
Record the cost of one served request of arm. Call with trace.final_.total_cost_usd.
fingerprint is the receipt’s; a mismatch with the current config counts the receipt in
ignored_stale and records nothing.
Sourcepub fn observe_health(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
errored: bool,
truncated: u64,
)
pub fn observe_health( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, errored: bool, truncated: u64, )
Tally one request of arm that errored and/or produced answers cut off at the output-token
limit (truncated counts attempts). Stale fingerprints are ignored: observe_cost already
counts the receipt once.
Sourcepub fn observe_outcome(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
session_id: &str,
verdict: Verdict,
) -> bool
pub fn observe_outcome( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, session_id: &str, verdict: Verdict, ) -> bool
Record a session’s task outcome. Returns whether it was counted.
Only the first clear outcome per session counts; later ones are ignored. The test pairs the k-th incumbent outcome with the k-th challenger outcome and treats sessions as independent draws, so several correlated outcomes from one long agent run would both overweight that session and break the independence the error bound rests on.
Pass is 1.0, Fail is 0.0, Abstain carries no information and is ignored.
Sourcepub fn reports(&self) -> Vec<Snapshot>
pub fn reports(&self) -> Vec<Snapshot>
A snapshot of every experiment, ordered by (tenant, route_ix).
Sourcepub async fn sync_from_backend(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
)
pub async fn sync_from_backend( &self, tenant: &str, route_ix: usize, fingerprint: &str, )
Read records appended to the shared log since the last sync and apply them locally.
Each call is cheap when there are no new records: one LRANGE that returns empty. Call this after every feedback event (within the feedback handler, not on the hot serve path) and from a background timer so all replicas stay ≤ 5 s stale.
No-op when no log backend is configured.
Sourcepub async fn record_outcome(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
arm: Arm,
session_id: &str,
verdict: Verdict,
cost_usd: f64,
) -> bool
pub async fn record_outcome( &self, tenant: &str, route_ix: usize, fingerprint: &str, arm: Arm, session_id: &str, verdict: Verdict, cost_usd: f64, ) -> bool
Record an outcome to the shared log (if configured), then sync from the log.
In single-replica mode (no log), falls back to the synchronous Self::observe_outcome.
cost_usd is stored in the log so other replicas can tally costs without needing the
full trace. Returns true when the outcome was counted.
Sourcepub async fn record_pair(
&self,
tenant: &str,
route_ix: usize,
fingerprint: &str,
x_inc: f64,
x_ch: f64,
incumbent_cost_usd: f64,
challenger_cost_usd: f64,
) -> bool
pub async fn record_pair( &self, tenant: &str, route_ix: usize, fingerprint: &str, x_inc: f64, x_ch: f64, incumbent_cost_usd: f64, challenger_cost_usd: f64, ) -> bool
Record a paired shadow result to the shared log (if configured), then sync.
In single-replica mode (no log), applies the pair directly and emits the status log.
Returns true if x_inc / x_ch are valid (finite, in [0, 1]) and the pair was
accepted by the underlying Experiment.
Sourcepub fn start_sync_task(self: &Arc<Self>, interval: Duration)
pub fn start_sync_task(self: &Arc<Self>, interval: Duration)
Spawn a background task that calls sync_from_backend for every known experiment at
most every interval (spec: ≤ 5 s). The task exits silently when the Arc becomes
the last reference. No-op when no log backend is configured.
Trait Implementations§
Source§impl Debug for ExperimentMonitor
impl Debug for ExperimentMonitor
Auto Trait Implementations§
impl !Freeze for ExperimentMonitor
impl !RefUnwindSafe for ExperimentMonitor
impl !UnwindSafe for ExperimentMonitor
impl Send for ExperimentMonitor
impl Sync for ExperimentMonitor
impl Unpin for ExperimentMonitor
impl UnsafeUnpin for ExperimentMonitor
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more