Expand description
Shadow A/B with an LLM judge.
A sampled fraction of routed traffic is replayed against the frontier model in the background, a judge scores the pair, and per-segment win rates are tallied so a regressed segment can raise an alert.
Re-exports§
pub use judge::JudgeVerdict;pub use scheduler::ShadowScheduler;
Modules§
- judge
- LLM-as-judge pairwise preference scoring.
- scheduler
- Shadow scheduler: samples a fraction of traffic and runs the frontier comparison + judge in the background.
Structs§
- ArcReexport
- A thread-safe reference-counting pointer. ‘Arc’ stands for ‘Atomically Reference Counted’.
- Registry
- Segment
Stats - Shadow
Config - Shadow
Enqueue - Glue type the proxy uses to enqueue shadow work without blocking.
- Shadow
Sample - One completed shadow comparison.
- WinRate
Aggregator - Win-rate aggregator: per-segment running tallies, plus regression flag.
Functions§
- log_
sample - Log a completed shadow sample.
- segment_
of - Bucket a request into a coarse aggregation segment, keyed on the opening words of its first user message.