rusty_agent_runtime/twin.rs
1//! The runtime digital twin (R0.10 wave 2): a deterministic re-execution
2//! environment for recorded runs.
3//!
4//! Exact replay ([`crate::replay`]) answers one question — "does the same
5//! graph reproduce the same evidence?" — and stops there. The twin answers
6//! the four questions plain replay cannot, per `docs/adaptation-design.md`:
7//!
8//! 1. **What happens under faults the recording never saw** — a
9//! [`FaultSchedule`]: a deterministic, seeded list of faults attached to
10//! a twin run. Injection lands at decision points and the effect
11//! boundary, never by patching code: the twin's scheduler reads the
12//! schedule the way the production scheduler reads the world.
13//! 2. **What happens under different concurrent interleavings** — schedule
14//! randomization: the parallel task set of each super-step is driven in
15//! a seeded order, so a concurrency policy is evaluated against the
16//! interleaving distribution, not the one schedule the recording
17//! happened to get. The journaled total order of evidence stays
18//! canonical; per-node latencies vary with the schedule.
19//! 3. **What happens if one decision changes** — a [`CounterfactualFork`]:
20//! apply one different legal action at one decision, continue with
21//! effects served from the recording where the decision leaves their
22//! inputs untouched, and compare with [`BranchDiff`]. With
23//! [`CounterfactualFork::then_act_with`] this is also R0.5's deferred
24//! *hybrid* replay: the recorded behavior up to the fork, a new policy
25//! afterward.
26//! 4. **What a candidate policy *would have* decided on the same evidence**
27//! — a shadow policy: the candidate decides, the floor acts, both
28//! journal as `PolicyDecision` events ([`DecisionRole::Shadow`] /
29//! [`DecisionRole::Acting`]) with their true propensities. Exploration
30//! by seeded draw makes the shadow a stochastic policy with known
31//! propensities: well-posed off-policy evidence by construction.
32//!
33//! # What the twin reuses (and does not re-implement)
34//!
35//! - **Determinism seams** ([`Clock`], [`RngSource`]): every timestamp,
36//! backoff jitter draw, interleaving permutation, and stochastic policy
37//! draw comes from the run's configured seams, so a twin run reproduces
38//! exactly from its seed and fixture.
39//! - **The effect journal's servable-kind vocabulary**
40//! ([`crate::replay::SERVABLE_KINDS`]) and integrity verification
41//! ([`ExactReplay`]'s boundary checks): the recorded world is loaded and
42//! verified by the replay machinery, then decomposed into work items.
43//! - **The retry policy gates** ([`crate::durable::classify_retry`]'s
44//! inputs via [`crate::durable::retry_legal_actions`]): the twin never
45//! re-implements the gates; the legal set a policy sees is computed by
46//! the same function the production scheduler uses.
47//! - **[`BranchDiff`]**: counterfactual comparison is journaled evidence,
48//! not a log line.
49//!
50//! # The honest edge
51//!
52//! The twin is a model of the runtime, not of the world. A counterfactual
53//! decision whose downstream effects are replay-servable — retry counts,
54//! backoff delays, timeout bounds, placement among equivalent workers,
55//! concurrency caps, checkpoint cadence — is exactly evaluable, because
56//! those decisions change *when and whether* effects execute, not what
57//! they return. A decision that would change an effect's *input* is
58//! unevaluable: the journal has no answer to a call the recorded world
59//! never received. The twin enforces the boundary structurally — policies
60//! choose among the closed [`DecisionAction`] set of one decision point,
61//! and no member of that set rewrites an effect's input — and refuses
62//! counterfactual forks outside the recomputed legal set. Every
63//! [`TwinReport`] states the bound in its required [`TwinReport::bound`]
64//! field (design open question 5: the constraint lives in the record, not
65//! in prose framing).
66//!
67//! Two modeling decisions, stated plainly:
68//!
69//! - **The world's baseline answer is the recording.** Each recorded
70//! effect is a work item whose truth is the journaled outcome. Re-served
71//! attempts (a retry re-issues the same bytes) receive the recorded
72//! answer again — the world answers the same bytes the same way. A
73//! recorded error therefore keeps failing across attempts: the recording
74//! is all the twin knows.
75//! - **The twin's scheduler is synchronous and simulated.** Work items
76//! execute on declared worker lanes; admission waits, timeout
77//! truncations, backoff delays, and lease-expiry discoveries are
78//! computed, not slept. What a policy *observes* (a timeout bound
79//! truncates the observed latency; the world's truth is untouched) is
80//! the Wave 1 rule carried into the twin. The graph executor and
81//! checkpointing are not driven: the twin's unit of evidence is the
82//! journal it writes, and checkpoint cadence evaluation stays with the
83//! R0.5 checkpoint machinery.
84
85use std::collections::BTreeMap;
86use std::fmt;
87use std::sync::Arc;
88
89use serde::{Deserialize, Serialize};
90use serde_json::{Map, Value};
91
92use crate::durable::{
93 backoff_delay_ms, backoff_delay_ms_with, resolve_retry_parameters, resolve_timeout_bound_ms,
94 retry_legal_actions, ErrorClass, MAX_RETRY_DELAY_MS,
95};
96use crate::error::{Result, RustyError};
97use crate::journal::{Clock, EventDraft, Journal, JournalSnapshot, RngSource};
98use crate::llm::Usage;
99use crate::record::{
100 DecisionAction, DecisionEvent, DecisionFamily, DecisionRole, Effect, EventStatus,
101 ExecutorPolicy, PayloadRef, PolicyVersion, RunEventKind,
102};
103use crate::replay::{BranchDiff, ExactReplay, LogicalClockParams, ReplayFixture, SERVABLE_KINDS};
104
105fn twin_error(message: impl Into<String>) -> RustyError {
106 // The twin's failures are replay-family failures: a run that cannot
107 // answer from its evidence must stop, not improvise.
108 RustyError::Replay(format!("twin: {}", message.into()))
109}
110
111/// The validity-boundary statement every [`TwinReport`] carries. Required,
112/// not optional: an auditor reads the twin's constraint from the record
113/// itself (design open question 5).
114pub const TWIN_REPORT_BOUND: &str = "the twin evaluates only decisions that change when/whether \
115 effects execute — retry counts, backoff delays, timeout bounds, placement among equivalent \
116 workers, concurrency caps, checkpoint cadence — never what effects return; a decision that \
117 would change an effect's input is unevaluable and was excluded or refused";
118
119/// The policy version a counterfactual fork journals under. A fork is not a
120/// policy — it is the experimenter's directed what-if, deterministic by
121/// construction — so its decisions are named as such rather than
122/// attributed to the floor or the candidate.
123pub const TWIN_FORK_POLICY_VERSION: &str = "twin-fork";
124
125/// The timeout ladder's minimum rung, in milliseconds. Below this, ordinary
126/// work aborts early — a correctness hazard no policy may cross (the same
127/// floor Wave 1 pre-registered). The constant lives with the family's
128/// parameter contract ([`crate::record::MIN_TIMEOUT_RUNG_MS`]); the twin
129/// re-exports it so the evaluation harness and the production read path
130/// enforce one bound.
131pub use crate::record::MIN_TIMEOUT_RUNG_MS;
132
133/// The default timeout ladder: discrete rungs between the minimum and the
134/// lease boundary ([`MAX_RETRY_DELAY_MS`], the queue's worst-case discovery
135/// latency for hung work — the floor's "no bound in force", modeled as the
136/// top rung exactly as Wave 1 modeled it).
137pub const DEFAULT_TIMEOUT_LADDER: [u64; 7] =
138 [100, 500, 1_000, 5_000, 30_000, 120_000, MAX_RETRY_DELAY_MS];
139
140/// The default concurrency ladder. The top rung is uncapped — the floor's
141/// stance when no concurrency policy is in force.
142pub const DEFAULT_CONCURRENCY_LADDER: [u32; 6] = [1, 2, 4, 8, 16, u32::MAX];
143
144// ---------------------------------------------------------------------------
145// Mechanism 1: fault injection.
146// ---------------------------------------------------------------------------
147
148/// One injectable fault. The taxonomy mirrors the world's failure modes the
149/// scheduler already classifies: each variant names the [`ErrorClass`] the
150/// injected failure is observed with.
151#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
152#[serde(tag = "fault", rename_all = "snake_case")]
153pub enum InjectedFault {
154 /// The worker dies mid-attempt. The attempt is lost and surfaces at the
155 /// lease boundary ([`MAX_RETRY_DELAY_MS`]) classified
156 /// [`ErrorClass::Unknown`] — the lease-expiry path, the dead-letter
157 /// queue's primary input.
158 WorkerCrash,
159
160 /// The attempt never completes on its own. With no timeout bound in
161 /// force it surfaces at the lease boundary; with a bound, at the bound,
162 /// classified [`ErrorClass::Timeout`] either way.
163 CalleeTimeout,
164
165 /// The callee asks to be slowed down: [`ErrorClass::RateLimited`] with
166 /// a `Retry-After` floor any policy's delay must respect.
167 RateLimited {
168 /// The callee-supplied `Retry-After` floor, in milliseconds.
169 retry_after_ms: u64,
170 },
171
172 /// The worker or callee is out of capacity:
173 /// [`ErrorClass::ResourceExhausted`]. Retryable — ideally on a
174 /// different worker, which is what makes this the placement family's
175 /// fault.
176 ResourceExhausted,
177}
178
179impl InjectedFault {
180 /// The class the scheduler observes for this fault.
181 pub fn error_class(&self) -> ErrorClass {
182 match self {
183 InjectedFault::WorkerCrash => ErrorClass::Unknown,
184 InjectedFault::CalleeTimeout => ErrorClass::Timeout,
185 InjectedFault::RateLimited { .. } => ErrorClass::RateLimited,
186 InjectedFault::ResourceExhausted => ErrorClass::ResourceExhausted,
187 }
188 }
189}
190
191/// Where a fault lands. Injection happens at decision points and the effect
192/// boundary — the two places the production scheduler reads the world.
193#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
194#[serde(tag = "anchor", rename_all = "snake_case")]
195pub enum FaultAnchor {
196 /// At the effect boundary: the `attempt`-th attempt (1-based) of the
197 /// effect recorded at `effect_seq` fails with the injected fault.
198 OnAttempt {
199 /// The recorded journal sequence number of the target effect.
200 effect_seq: u64,
201 /// The 1-based attempt ordinal the fault fires on.
202 attempt: u32,
203 },
204
205 /// At a decision point: when the scheduler makes the retry decision
206 /// with sequence `decision_seq`, the in-flight attempt is observed
207 /// through the injected fault instead of its real outcome (a crash
208 /// here is the lease-expiry path: the worker died, the attempt is
209 /// lost, the failure classifies [`ErrorClass::Unknown`]). Fires only
210 /// at retry decision points — the scheduler's fail-task path, where
211 /// the production scheduler would observe a lost attempt; an anchor
212 /// that never coincides with one never fires, and the report's
213 /// fired/declared counts say so.
214 AtDecision {
215 /// The twin run's decision sequence number (`d{n}`) to fire at.
216 decision_seq: u64,
217 },
218
219 /// A provider rate-limit window: every attempt of every effect recorded
220 /// in `from_seq..=to_seq` is answered with the injected fault — the
221 /// provider-side degradation a recording from a healthy window never
222 /// contains.
223 Window {
224 /// First recorded effect sequence covered by the window.
225 from_seq: u64,
226 /// Last recorded effect sequence covered by the window (inclusive).
227 to_seq: u64,
228 },
229
230 /// Resource exhaustion on one worker: every attempt placed on `worker`
231 /// fails. The placement family's evaluation fault: work steered onto
232 /// the degraded worker pays for it.
233 OnWorker {
234 /// The degraded worker's identity (a member of the run's pool).
235 worker: String,
236 },
237}
238
239/// One entry of a [`FaultSchedule`]: an anchor plus the fault that fires
240/// there.
241#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
242pub struct FaultInjection {
243 /// Where the fault lands.
244 #[serde(flatten)]
245 pub anchor: FaultAnchor,
246
247 /// What the world does there.
248 #[serde(flatten)]
249 pub fault: InjectedFault,
250}
251
252/// A deterministic, seeded fault schedule attached to a twin run.
253///
254/// The schedule is *data*, declared before the run: the twin's scheduler
255/// reads it the way the production scheduler reads the world, and two runs
256/// over the same fixture with the same schedule and seed fire exactly the
257/// same faults at exactly the same points. Serializable so a schedule can
258/// be committed alongside the fixture it probes — the evaluation's
259/// reproducibility artifact.
260#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
261pub struct FaultSchedule {
262 /// The seed the schedule was authored under (carried for provenance;
263 /// the run's own seed governs draws).
264 pub seed: u64,
265
266 /// The declared injections, consulted in declaration order: the first
267 /// anchor matching an attempt or decision point wins.
268 #[serde(default, skip_serializing_if = "Vec::is_empty")]
269 pub injections: Vec<FaultInjection>,
270}
271
272impl FaultSchedule {
273 /// An empty schedule under `seed`: the recorded world, unfaulted.
274 pub fn new(seed: u64) -> Self {
275 Self {
276 seed,
277 injections: Vec::new(),
278 }
279 }
280
281 /// Builder-style: declare one more injection.
282 pub fn with_injection(mut self, anchor: FaultAnchor, fault: InjectedFault) -> Self {
283 self.injections.push(FaultInjection { anchor, fault });
284 self
285 }
286
287 /// The fault firing on attempt `attempt` of the effect recorded at
288 /// `effect_seq` when placed on `worker`, if any. Consulted at the
289 /// effect boundary, once per attempt.
290 pub fn fault_for_attempt(
291 &self,
292 effect_seq: u64,
293 attempt: u32,
294 worker: &str,
295 ) -> Option<&InjectedFault> {
296 self.injections
297 .iter()
298 .find(|injection| match &injection.anchor {
299 FaultAnchor::OnAttempt {
300 effect_seq: seq,
301 attempt: ord,
302 } => *seq == effect_seq && *ord == attempt,
303 FaultAnchor::Window { from_seq, to_seq } => {
304 *from_seq <= effect_seq && effect_seq <= *to_seq
305 }
306 FaultAnchor::OnWorker { worker: w } => w == worker,
307 FaultAnchor::AtDecision { .. } => false,
308 })
309 .map(|injection| &injection.fault)
310 }
311
312 /// The fault firing at the retry decision with sequence `decision_seq`,
313 /// if any. Consulted at the scheduler's fail-task decision point only.
314 pub fn fault_for_decision(&self, decision_seq: u64) -> Option<&InjectedFault> {
315 self.injections
316 .iter()
317 .find(|injection| match &injection.anchor {
318 FaultAnchor::AtDecision { decision_seq: seq } => *seq == decision_seq,
319 _ => false,
320 })
321 .map(|injection| &injection.fault)
322 }
323}
324
325// ---------------------------------------------------------------------------
326// Mechanism 4's other half: the policies. (The shadow pair itself is the
327// run machinery below.)
328// ---------------------------------------------------------------------------
329
330/// What a policy sees at one decision point: the family, the journaled
331/// feature map, and the closed legal-action set computed by the runtime's
332/// own gates. A policy chooses among `legal_actions` members and declares
333/// its propensity — never a free-form output.
334#[derive(Debug)]
335pub struct DecisionContext<'a> {
336 /// Which executor decision this is.
337 pub family: DecisionFamily,
338
339 /// The observation assembled at the decision point.
340 pub features: &'a Map<String, Value>,
341
342 /// Every action legal at decision time.
343 pub legal_actions: &'a [DecisionAction],
344}
345
346/// A policy the twin can evaluate: the floor, or a candidate scoring the
347/// same features.
348///
349/// `decide` receives one seeded `draw` per invocation so stochastic
350/// candidates explore by seeded draw — a stochastic policy with known
351/// propensities, which is what makes its journaled shadow decisions
352/// well-posed off-policy evidence. Implementations must return a member of
353/// `legal_actions` and a propensity in `(0, 1]`; the twin rejects either
354/// violation at the boundary rather than journaling dishonest evidence.
355pub trait TwinPolicy: fmt::Debug + Send + Sync {
356 /// The version the policy's decisions journal under.
357 fn version(&self) -> PolicyVersion;
358
359 /// Choose an action from `ctx.legal_actions` and declare its
360 /// propensity. `draw` is a seeded uniform from `[0, 1)`.
361 fn decide(&self, ctx: &DecisionContext<'_>, draw: f64) -> (DecisionAction, f64);
362
363 /// The re-queue delay after a `Retry` decision, in milliseconds. The
364 /// closed action set carries no delay member — the delay is
365 /// policy-parameterized within declared bounds, per the family's
366 /// contract — so it is a method, not an action. Defaults to the
367 /// floor's full-jitter exponential backoff.
368 fn retry_delay_ms(&self, attempt: u32, draw: f64) -> u64 {
369 backoff_delay_ms(attempt, draw)
370 }
371
372 /// The re-queue delay with the failure class in scope (R0.10 wave 3):
373 /// learned policies tune the schedule per class
374 /// ([`ParameterizedPolicy`]); policies that do not override this keep
375 /// [`TwinPolicy::retry_delay_ms`]'s class-blind behavior. The default
376 /// delegates, so every pre-wave-3 policy is unchanged.
377 fn retry_delay_ms_for(&self, class: Option<ErrorClass>, attempt: u32, draw: f64) -> u64 {
378 let _ = class;
379 self.retry_delay_ms(attempt, draw)
380 }
381}
382
383/// The `static-v0` floor: the behavior every pre-learning run had, and the
384/// baseline every candidate is evaluated against.
385///
386/// Per family: **retry** retries whenever the gates leave a `Retry` member
387/// in the legal set (the legal set is computed by
388/// [`retry_legal_actions`], so the floor's stance is exactly
389/// `classify_retry`'s); **timeout** imposes no bound — modeled as the
390/// ladder's top rung, the lease boundary, Wave 1's `LEASE_BOUND_MS`;
391/// **placement** takes the first eligible worker (the static-pool stance:
392/// no re-placement intelligence); **concurrency** is uncapped. Every
393/// selection has propensity 1.0: the floor is deterministic.
394#[derive(Debug, Default, Clone, Copy)]
395pub struct StaticFloor;
396
397impl TwinPolicy for StaticFloor {
398 fn version(&self) -> PolicyVersion {
399 PolicyVersion::new(PolicyVersion::STATIC_V0)
400 }
401
402 fn decide(&self, ctx: &DecisionContext<'_>, _draw: f64) -> (DecisionAction, f64) {
403 let selected = match ctx.family {
404 DecisionFamily::Retry => ctx
405 .legal_actions
406 .iter()
407 .find(|action| matches!(action, DecisionAction::Retry { .. }))
408 .cloned()
409 .unwrap_or(DecisionAction::Abort),
410 // The floor's "no bound in force" and "unlimited" stances are
411 // the ladder tops; placement's static pool is the first
412 // eligible worker; the checkpoint floor writes.
413 DecisionFamily::Timeout
414 | DecisionFamily::Concurrency
415 | DecisionFamily::CheckpointPlacement => ctx
416 .legal_actions
417 .last()
418 .cloned()
419 .unwrap_or(DecisionAction::Abort),
420 DecisionFamily::WorkerPlacement => ctx
421 .legal_actions
422 .first()
423 .cloned()
424 .unwrap_or(DecisionAction::Abort),
425 };
426 (selected, 1.0)
427 }
428}
429
430/// The application loop as a twin policy (R0.10 wave 3): an
431/// [`ExecutorPolicy`] — the registry body a promoted candidate produced —
432/// deciding every family with its parameters, so the twin prices the
433/// exact behavior a bound run would execute.
434///
435/// Per family: **retry** applies the per-class backoff table and narrowed
436/// attempt budget ([`resolve_retry_parameters`]) — the gates stay the
437/// runtime's own (the legal set is computed before the policy sees it, so
438/// no parameter set can route around them); **timeout** applies the
439/// per-callee/default bound ([`resolve_timeout_bound_ms`]) honored by the
440/// smallest ladder rung that covers it; **placement, concurrency, and
441/// checkpoint placement decide exactly as the floor** — those families
442/// have no parameter contract ([`ExecutorPolicy::with_family_parameters`]
443/// rejects them), so a registered policy can never carry their parameters
444/// and there is nothing to apply. Every selection is deterministic
445/// (propensity 1.0): learned exploration belongs to shadow candidates
446/// with declared propensities, not to a promoted body.
447///
448/// Fail-closed at two layers, matching the production read path:
449/// construction refuses an invalid policy ([`ExecutorPolicy::validate`] —
450/// an out-of-envelope body cannot even be evaluated), and the
451/// `resolve_*` functions re-check at every decision point, so a policy
452/// that somehow reaches a decision invalid steers nothing and the floor
453/// decides.
454#[derive(Debug, Clone)]
455pub struct ParameterizedPolicy {
456 policy: ExecutorPolicy,
457 version: PolicyVersion,
458}
459
460impl ParameterizedPolicy {
461 /// A policy deciding with `policy`'s parameters, journaling under
462 /// `version` (the body the registry holds for that version —
463 /// mismatched pairs are the caller's integrity failure to make).
464 pub fn new(policy: ExecutorPolicy, version: PolicyVersion) -> Result<Self> {
465 policy.validate().map_err(|e| {
466 twin_error(format!(
467 "refusing to evaluate policy `{version}`: its parameters fall outside the \
468 envelope ({e}) — an invalid body cannot act, in evaluation or in production"
469 ))
470 })?;
471 Ok(Self { policy, version })
472 }
473
474 /// The parameters this policy decides with.
475 pub fn policy(&self) -> &ExecutorPolicy {
476 &self.policy
477 }
478
479 /// Read one error class out of a decision's journaled features (the
480 /// retry family's `failure_class` key — absent or unparseable yields
481 /// `Unknown`, the DLQ's conservative default).
482 fn failure_class(ctx: &DecisionContext<'_>) -> ErrorClass {
483 ctx.features
484 .get("failure_class")
485 .and_then(|value| serde_json::from_value(value.clone()).ok())
486 .unwrap_or(ErrorClass::Unknown)
487 }
488
489 /// Read a `u64` feature by key, when present.
490 fn u64_feature(ctx: &DecisionContext<'_>, key: &str) -> Option<u64> {
491 ctx.features.get(key).and_then(serde_json::Value::as_u64)
492 }
493}
494
495impl TwinPolicy for ParameterizedPolicy {
496 fn version(&self) -> PolicyVersion {
497 self.version.clone()
498 }
499
500 fn decide(&self, ctx: &DecisionContext<'_>, _draw: f64) -> (DecisionAction, f64) {
501 let selected = match ctx.family {
502 DecisionFamily::Retry => {
503 let class = Self::failure_class(ctx);
504 let attempt = Self::u64_feature(ctx, "attempt").unwrap_or(1) as u32;
505 let budget = Self::u64_feature(ctx, "max_attempts").unwrap_or(3) as u32;
506 let resolved = resolve_retry_parameters(&self.policy, class, budget);
507 // The policy's budget narrows the decision, never the legal
508 // set: retrying stays legal (the runtime's gates said so),
509 // the policy declines it past its learned budget.
510 if attempt < resolved.max_attempts {
511 ctx.legal_actions
512 .iter()
513 .find(|action| matches!(action, DecisionAction::Retry { .. }))
514 .cloned()
515 .unwrap_or(DecisionAction::Abort)
516 } else {
517 DecisionAction::Abort
518 }
519 }
520 DecisionFamily::Timeout => {
521 // The twin journals the callee identity as `node` (the node
522 // that issued the call); the production emission point
523 // journals `callee`. Both name the same identity.
524 let callee = ctx
525 .features
526 .get("callee")
527 .or_else(|| ctx.features.get("node"))
528 .and_then(serde_json::Value::as_str);
529 let bound = resolve_timeout_bound_ms(&self.policy, callee);
530 match bound {
531 Some(bound) => ctx
532 .legal_actions
533 .iter()
534 .find(|action| matches!(action, DecisionAction::SetTimeout { millis } if *millis >= bound))
535 .cloned()
536 .or_else(|| ctx.legal_actions.last().cloned())
537 .unwrap_or(DecisionAction::Abort),
538 // No bound in force: the floor's stance, the top rung.
539 None => ctx.legal_actions.last().cloned().unwrap_or(DecisionAction::Abort),
540 }
541 }
542 // Shadow-only families decide exactly as the floor: no
543 // parameter contract exists, so there is nothing to apply.
544 DecisionFamily::Concurrency | DecisionFamily::CheckpointPlacement => ctx
545 .legal_actions
546 .last()
547 .cloned()
548 .unwrap_or(DecisionAction::Abort),
549 DecisionFamily::WorkerPlacement => ctx
550 .legal_actions
551 .first()
552 .cloned()
553 .unwrap_or(DecisionAction::Abort),
554 };
555 (selected, 1.0)
556 }
557
558 fn retry_delay_ms_for(&self, class: Option<ErrorClass>, attempt: u32, draw: f64) -> u64 {
559 let retry = &self.policy.retry;
560 if retry.validate().is_err() {
561 return backoff_delay_ms(attempt, draw);
562 }
563 let schedule = class
564 .and_then(|class| retry.per_class.as_ref()?.get(&class).copied())
565 .unwrap_or_else(|| retry.flat());
566 backoff_delay_ms_with(attempt, draw, schedule.base_delay_ms, schedule.max_delay_ms)
567 }
568}
569
570// ---------------------------------------------------------------------------
571// The recorded world: a journaled run decomposed into work items.
572// ---------------------------------------------------------------------------
573
574/// What the recorded world did for one effect: the journaled outcome,
575/// resolved out of the snapshot's payload references.
576#[derive(Debug, Clone)]
577pub struct RecordedAnswer {
578 /// How the recorded call ended (`Interrupted` is rejected at load:
579 /// servable calls never produce it).
580 pub status: EventStatus,
581
582 /// The recorded response payload.
583 pub output: Option<Value>,
584
585 /// The recorded latency in milliseconds (0 when unmeasured).
586 pub latency_ms: u64,
587
588 /// The recorded token usage, when reported.
589 pub tokens: Option<Usage>,
590
591 /// The recorded cost in USD, when known.
592 pub cost_usd: Option<f64>,
593}
594
595/// One recorded effect as a unit of twin work: the request the world
596/// answered, the answer it gave, the effect classification the gates read,
597/// and the super-step the journal places it in.
598#[derive(Debug, Clone)]
599pub struct TwinWorkItem {
600 /// The recorded event's journal sequence number (fault anchors name
601 /// effects by it).
602 pub recorded_seq: u64,
603
604 /// The recorded effect kind (a member of [`SERVABLE_KINDS`]).
605 pub kind: RunEventKind,
606
607 /// The node that issued the call, when journaled.
608 pub node_id: Option<String>,
609
610 /// The declared effect class — the retry gate's input.
611 pub effect: Effect,
612
613 /// The request, exactly as journaled. Retried attempts re-issue these
614 /// bytes; the twin never rewrites them.
615 pub request: Value,
616
617 /// What the recorded world answered.
618 pub answer: RecordedAnswer,
619
620 /// The super-step the recording placed this effect in (0 when the
621 /// journal carries no super-step structure).
622 pub step: u64,
623}
624
625/// The recorded world a twin run re-executes against: the fixture's
626/// servable effects as work items, grouped by super-step.
627#[derive(Debug, Clone)]
628pub struct TwinWorld {
629 items: Vec<TwinWorkItem>,
630 /// Distinct super-step indexes in ascending order.
631 steps: Vec<u64>,
632}
633
634impl TwinWorld {
635 /// Decompose an integrity-verified snapshot into work items. Effects
636 /// journaled between `SuperStepStart` boundaries inherit that step;
637 /// everything before the first boundary (or in a journal with no
638 /// boundaries) lands in step 0.
639 pub fn from_snapshot(snapshot: &JournalSnapshot) -> Result<Self> {
640 let resolve = |payload: &PayloadRef| -> Option<Value> {
641 match payload {
642 PayloadRef::Inline(value) => Some(value.clone()),
643 PayloadRef::Artifact(reference) => {
644 snapshot.artifacts.get(&reference.sha256).cloned()
645 }
646 }
647 };
648 let mut items = Vec::new();
649 let mut steps = Vec::new();
650 let mut current_step = 0u64;
651 for event in &snapshot.events {
652 if event.kind == RunEventKind::SuperStepStart {
653 current_step = event
654 .input
655 .as_ref()
656 .and_then(&resolve)
657 .and_then(|input| input.get("step").and_then(Value::as_u64))
658 .unwrap_or(current_step + 1);
659 steps.push(current_step);
660 continue;
661 }
662 if !SERVABLE_KINDS.contains(&event.kind) {
663 continue;
664 }
665 if event.status == EventStatus::Interrupted {
666 return Err(twin_error(format!(
667 "recorded {:?} at seq {} has status `interrupted`, which servable calls \
668 never produce — the journal is inconsistent",
669 event.kind, event.seq
670 )));
671 }
672 let request = event.input.as_ref().and_then(&resolve).ok_or_else(|| {
673 twin_error(format!(
674 "recorded {:?} at seq {} has no input payload; the twin needs the request \
675 the world answered",
676 event.kind, event.seq
677 ))
678 })?;
679 let answer = RecordedAnswer {
680 status: event.status,
681 output: event.output.as_ref().and_then(&resolve),
682 latency_ms: event.latency_ms.unwrap_or(0),
683 tokens: event.tokens,
684 cost_usd: event.cost_usd,
685 };
686 items.push(TwinWorkItem {
687 recorded_seq: event.seq,
688 kind: event.kind,
689 node_id: event.node_id.clone(),
690 effect: event.effect,
691 request,
692 answer,
693 step: current_step,
694 });
695 }
696 if steps.is_empty() && !items.is_empty() {
697 steps.push(0);
698 }
699 Ok(Self { items, steps })
700 }
701
702 /// Every work item, in recorded sequence order.
703 pub fn items(&self) -> &[TwinWorkItem] {
704 &self.items
705 }
706
707 /// The super-step indexes the recording contains, ascending.
708 pub fn steps(&self) -> &[u64] {
709 &self.steps
710 }
711
712 /// The work items of one super-step, in recorded (canonical) order.
713 pub fn items_in(&self, step: u64) -> Vec<&TwinWorkItem> {
714 self.items.iter().filter(|item| item.step == step).collect()
715 }
716}
717
718// ---------------------------------------------------------------------------
719// Run configuration and outcomes.
720// ---------------------------------------------------------------------------
721
722/// How the parallel task set of each super-step is ordered.
723#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
724#[serde(rename_all = "snake_case")]
725pub enum Interleaving {
726 /// Recorded order. The baseline every randomized run compares against.
727 #[default]
728 Canonical,
729
730 /// A seeded permutation drawn from the run's [`RngSource`] — mechanism
731 /// 2: the same evidence re-driven under one point of the interleaving
732 /// distribution. Journaled event order stays canonical; admission
733 /// waits, and therefore per-node latencies, follow the drawn order.
734 Seeded,
735}
736
737/// The parameters of one twin run: determinism seams, the fault schedule,
738/// the acting policy, an optional shadow, the decision ladders, and the
739/// worker pool.
740///
741/// Two runs over the same [`Twin`] with equal configs produce byte-identical
742/// journals — the twin's whole claim, proven in the test suite across
743/// repeated runs and across process invocations (the checked-in golden).
744pub struct TwinRunConfig {
745 /// The run's seed: RNG draws, interleaving permutations, and stochastic
746 /// policy draws all derive from it.
747 pub seed: u64,
748
749 /// The logical clock the run's journal timestamps from.
750 pub clock: LogicalClockParams,
751
752 /// The fault schedule the scheduler reads. Empty means the recorded
753 /// world, unfaulted.
754 pub faults: FaultSchedule,
755
756 /// The policy whose actions execute. The floor by default.
757 pub acting: Arc<dyn TwinPolicy>,
758
759 /// A candidate scoring the same features without acting (mechanism 4).
760 pub shadow: Option<Arc<dyn TwinPolicy>>,
761
762 /// The timeout family's legal rungs, ascending. The top rung models
763 /// "no bound in force"; keep it at [`MAX_RETRY_DELAY_MS`] for
764 /// comparability with Wave 1.
765 pub timeout_ladder: Vec<u64>,
766
767 /// The concurrency family's legal rungs, ascending; `u32::MAX` is
768 /// uncapped.
769 pub concurrency_ladder: Vec<u32>,
770
771 /// The worker pool placement ranks. Equivalence is a precondition: the
772 /// pool declares the workers interchangeable for this workload.
773 pub workers: Vec<String>,
774
775 /// The retry attempt budget (the floor's own is 3).
776 pub max_attempts: u32,
777
778 /// Canonical or seeded ordering of each super-step's task set.
779 pub interleaving: Interleaving,
780}
781
782impl TwinRunConfig {
783 /// A baseline configuration: the floor acting, no shadow, no faults,
784 /// canonical order, default ladders, one worker, the floor's attempt
785 /// budget.
786 pub fn new(seed: u64) -> Self {
787 Self {
788 seed,
789 clock: LogicalClockParams {
790 start_ms: 1_700_000_000_000,
791 tick_ms: 10,
792 },
793 faults: FaultSchedule::new(seed),
794 acting: Arc::new(StaticFloor),
795 shadow: None,
796 timeout_ladder: DEFAULT_TIMEOUT_LADDER.to_vec(),
797 concurrency_ladder: DEFAULT_CONCURRENCY_LADDER.to_vec(),
798 workers: vec!["worker-0".to_owned()],
799 max_attempts: 3,
800 interleaving: Interleaving::Canonical,
801 }
802 }
803
804 /// Builder-style: attach a fault schedule.
805 pub fn with_faults(mut self, faults: FaultSchedule) -> Self {
806 self.faults = faults;
807 self
808 }
809
810 /// Builder-style: act with `policy` instead of the floor.
811 pub fn with_acting(mut self, policy: Arc<dyn TwinPolicy>) -> Self {
812 self.acting = policy;
813 self
814 }
815
816 /// Builder-style: shadow the acting policy with `candidate`.
817 pub fn with_shadow(mut self, candidate: Arc<dyn TwinPolicy>) -> Self {
818 self.shadow = Some(candidate);
819 self
820 }
821
822 /// Builder-style: declare the worker pool.
823 pub fn with_workers(mut self, workers: impl IntoIterator<Item = impl Into<String>>) -> Self {
824 self.workers = workers.into_iter().map(Into::into).collect();
825 self
826 }
827
828 /// Builder-style: override the retry attempt budget.
829 pub fn with_max_attempts(mut self, max_attempts: u32) -> Self {
830 self.max_attempts = max_attempts;
831 self
832 }
833
834 /// Builder-style: order each super-step's task set by seeded draw.
835 pub fn with_seeded_interleaving(mut self) -> Self {
836 self.interleaving = Interleaving::Seeded;
837 self
838 }
839}
840
841impl fmt::Debug for TwinRunConfig {
842 fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
843 f.debug_struct("TwinRunConfig")
844 .field("seed", &self.seed)
845 .field("clock", &self.clock)
846 .field("faults", &self.faults)
847 .field("acting", &self.acting.version())
848 .field("shadow", &self.shadow.as_ref().map(|p| p.version()))
849 .field("workers", &self.workers)
850 .field("max_attempts", &self.max_attempts)
851 .field("interleaving", &self.interleaving)
852 .finish()
853 }
854}
855
856/// Aggregate outcome metrics of one twin run. Simulated, deterministic, and
857/// directly comparable across arms — the same accounting Wave 1 prices.
858#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
859pub struct TwinMetrics {
860 /// Work items the run contained.
861 pub items: usize,
862
863 /// Items that reached a successful terminal attempt.
864 pub completed: usize,
865
866 /// Items that exhausted the attempt budget on a retryable failure.
867 pub dead_lettered: usize,
868
869 /// Items that failed permanently (the effect or class gate).
870 pub failed: usize,
871
872 /// Total attempts executed across all items.
873 pub attempts: u64,
874
875 /// Summed recorded cost in USD across served attempts.
876 pub cost_usd: f64,
877
878 /// Simulated wall time of the whole run, in milliseconds.
879 pub wall_time_ms: u64,
880
881 /// Nearest-rank p50 of per-item elapsed time (first ready to terminal),
882 /// in milliseconds.
883 pub item_latency_p50_ms: u64,
884
885 /// Nearest-rank p95 of per-item elapsed time, in milliseconds.
886 pub item_latency_p95_ms: u64,
887}
888
889/// A case the twin excluded or flagged under the honest edge. Structural
890/// prevention covers most of the boundary (no legal action rewrites an
891/// effect's input); these are the cases that still need a name in the
892/// report.
893#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
894#[serde(tag = "case", rename_all = "snake_case")]
895pub enum UnevaluableCase {
896 /// A fault fired on work whose declared effect is not freely
897 /// repeatable: the effect gate fails the item on the first fault, so
898 /// retry counterfactuals past that point have no reachable world.
899 /// Flagged rather than silent: a schedule author should read that the
900 /// gate, not the policy, decided the item's future.
901 GatedEffect {
902 /// The recorded sequence number of the gated effect.
903 effect_seq: u64,
904 },
905
906 /// A counterfactual fork named an action outside the legal set
907 /// recomputed at its decision point — including every input-changing
908 /// action, which no legal set ever contains. Always a refusal, never a
909 /// journaled run.
910 IllegalForkAction {
911 /// The decision the fork named.
912 decision_seq: u64,
913 /// The action that was refused.
914 action: DecisionAction,
915 },
916
917 /// A counterfactual fork named a decision the run never reaches.
918 UnknownDecision {
919 /// The decision sequence that does not exist in this run.
920 decision_seq: u64,
921 },
922}
923
924/// The report every twin run emits: what was evaluated, what was excluded,
925/// and the bound the evaluation holds to. The validity boundary is a
926/// required field of the payload, not prose about it.
927#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
928pub struct TwinReport {
929 /// The twin run's id.
930 pub run_id: String,
931
932 /// The seed the run drew from.
933 pub seed: u64,
934
935 /// Faults the schedule declared.
936 pub faults_declared: usize,
937
938 /// Faults that actually fired. Less than declared means anchors that
939 /// never coincided with an attempt or retry decision — visible, never
940 /// silent.
941 pub faults_fired: usize,
942
943 /// Decisions journaled by the acting policy (shadow decisions add
944 /// their own events but are not separate decision points).
945 pub decisions: usize,
946
947 /// Decision points whose outcomes are exactly evaluable: replay-servable
948 /// or fault-injected downstream. In this model every journaled decision
949 /// is evaluable by construction; the field exists so a future mechanism
950 /// that is not must subtract here.
951 pub evaluable_decisions: usize,
952
953 /// Decision sequences where the shadow's selection differed from the
954 /// acting policy's — the points a counterfactual branch estimates the
955 /// candidate's outcome from.
956 pub shadow_divergences: Vec<u64>,
957
958 /// Cases excluded or flagged under the honest edge.
959 pub excluded: Vec<UnevaluableCase>,
960
961 /// The validity-boundary statement. Always [`TWIN_REPORT_BOUND`];
962 /// carried as a field so the constraint travels with the evidence.
963 pub bound: String,
964}
965
966/// The result of one twin run: the journaled evidence, the decision stream,
967/// the metrics, and the report.
968#[derive(Debug)]
969pub struct TwinOutcome {
970 /// The twin run's journal snapshot (super-step boundaries, served and
971 /// faulted effect events, and every `PolicyDecision`).
972 pub journal: JournalSnapshot,
973
974 /// Every decision the run journaled — acting and shadow interleaved by
975 /// decision point, in decision order.
976 pub decisions: Vec<DecisionEvent>,
977
978 /// Aggregate metrics.
979 pub metrics: TwinMetrics,
980
981 /// The validity report.
982 pub report: TwinReport,
983}
984
985/// A directed what-if: at `decision_seq`, apply `action` instead of the
986/// acting policy's choice, then continue.
987#[derive(Debug)]
988pub struct CounterfactualFork {
989 /// The twin run's decision sequence number to fork at (a report's
990 /// `shadow_divergences` are the natural candidates).
991 pub decision_seq: u64,
992
993 /// The different legal action to apply. Must be a member of the legal
994 /// set recomputed at that decision point — the twin refuses anything
995 /// else, which is where the honest edge bites: no legal set contains an
996 /// input-changing action.
997 pub action: DecisionAction,
998
999 /// Hybrid replay (R0.5's deferred mode): after the fork, act with this
1000 /// policy instead of the run's original acting policy. `None` keeps the
1001 /// original acting policy — a single-decision what-if.
1002 pub then_act_with: Option<Arc<dyn TwinPolicy>>,
1003}
1004
1005/// The result of a counterfactual branch: the branch's own outcome plus its
1006/// [`BranchDiff`] against the baseline — fork-plus-diff, comparison as
1007/// journaled evidence.
1008#[derive(Debug)]
1009pub struct CounterfactualBranch {
1010 /// The branch run's outcome.
1011 pub outcome: TwinOutcome,
1012
1013 /// The baseline's outcome the branch was diffed against.
1014 pub baseline: TwinOutcome,
1015
1016 /// The logical diff: the fork's divergence point, the work each side
1017 /// did after it, step-level differences, and per-side totals.
1018 pub diff: BranchDiff,
1019}
1020
1021// ---------------------------------------------------------------------------
1022// The twin.
1023// ---------------------------------------------------------------------------
1024
1025/// A deterministic re-execution environment over one recorded run.
1026///
1027/// Construction verifies the journal's integrity through the replay
1028/// module's boundary ([`ExactReplay::new`]) and decomposes the servable
1029/// effects into the world the scheduler re-drives. The typical flow:
1030///
1031/// 1. `let twin = Twin::from_fixture(&fixture)?;`
1032/// 2. `let baseline = twin.run(&TwinRunConfig::new(seed))?;` — the
1033/// recorded world under the floor;
1034/// 3. probe it: `twin.run(&config.with_faults(schedule))`,
1035/// `twin.run_interleavings(8, &config.with_seeded_interleaving())`,
1036/// `twin.run(&config.with_shadow(candidate))`, or
1037/// `twin.counterfactual(&config, fork)`.
1038#[derive(Debug)]
1039pub struct Twin {
1040 world: TwinWorld,
1041 run_id: String,
1042 thread_id: String,
1043}
1044
1045impl Twin {
1046 /// A twin over an integrity-verified snapshot. Resumed-run journals are
1047 /// rejected with the same boundary exact replay applies: their evidence
1048 /// begins mid-run against checkpointed state the journal does not
1049 /// carry.
1050 pub fn from_snapshot(snapshot: JournalSnapshot) -> Result<Self> {
1051 // Integrity verification (head hash, artifacts, references, the
1052 // resumed-run boundary) is the replay module's contract, consumed —
1053 // not re-implemented.
1054 ExactReplay::new(snapshot.clone())?;
1055 let world = TwinWorld::from_snapshot(&snapshot)?;
1056 Ok(Self {
1057 world,
1058 run_id: snapshot.run_id,
1059 thread_id: snapshot.thread_id,
1060 })
1061 }
1062
1063 /// A twin over a portable fixture's journal: a recorded production run
1064 /// becomes a twin case by export, unmodified.
1065 pub fn from_fixture(fixture: &ReplayFixture) -> Result<Self> {
1066 Self::from_snapshot(fixture.journal.clone())
1067 }
1068
1069 /// The recorded world this twin re-executes against.
1070 pub fn world(&self) -> &TwinWorld {
1071 &self.world
1072 }
1073
1074 /// The twin run id for `seed` — deterministic, so equal configs share
1075 /// an id and their journals compare byte-for-byte.
1076 fn twin_run_id(&self, seed: u64) -> String {
1077 format!("twin:{}:{seed:016x}", self.run_id)
1078 }
1079
1080 /// Re-execute the recorded world under `config`.
1081 pub fn run(&self, config: &TwinRunConfig) -> Result<TwinOutcome> {
1082 self.run_inner(config, None)
1083 }
1084
1085 /// Re-run the recorded world under `runs` seeded interleavings
1086 /// (mechanism 2). Run `k` draws its permutation and jitter from
1087 /// `config.seed` mixed with `k`, so the interleaving distribution is
1088 /// sampled deterministically and each point of it reproduces exactly.
1089 /// The journaled total order of evidence is canonical in every run —
1090 /// what varies is admission order and therefore per-node latencies.
1091 pub fn run_interleavings(
1092 &self,
1093 runs: usize,
1094 config: &TwinRunConfig,
1095 ) -> Result<Vec<TwinOutcome>> {
1096 (0..runs)
1097 .map(|k| {
1098 let run_config = TwinRunConfig {
1099 seed: config.seed ^ (k as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15),
1100 clock: config.clock,
1101 faults: config.faults.clone(),
1102 acting: config.acting.clone(),
1103 shadow: config.shadow.clone(),
1104 timeout_ladder: config.timeout_ladder.clone(),
1105 concurrency_ladder: config.concurrency_ladder.clone(),
1106 workers: config.workers.clone(),
1107 max_attempts: config.max_attempts,
1108 interleaving: Interleaving::Seeded,
1109 };
1110 self.run(&run_config)
1111 })
1112 .collect()
1113 }
1114
1115 /// Fork the recorded run at one decision: `baseline` behavior up to the
1116 /// fork, `fork.action` applied at it, then `fork.then_act_with` (or the
1117 /// original acting policy) afterward — with the same fault schedule
1118 /// still in force.
1119 ///
1120 /// Refuses, with [`UnevaluableCase`] naming the case, when the fork's
1121 /// action is outside the legal set recomputed at its decision point
1122 /// (the honest edge made structural) or when the run never reaches the
1123 /// named decision.
1124 pub fn counterfactual(
1125 &self,
1126 baseline: &TwinRunConfig,
1127 fork: CounterfactualFork,
1128 ) -> Result<CounterfactualBranch> {
1129 let base_outcome = self.run(baseline)?;
1130 let branch_outcome = self.run_inner(baseline, Some(&fork))?;
1131 let diff = BranchDiff::between(&base_outcome.journal, &branch_outcome.journal);
1132 Ok(CounterfactualBranch {
1133 outcome: branch_outcome,
1134 baseline: base_outcome,
1135 diff,
1136 })
1137 }
1138
1139 /// The shared run machinery. With `fork = Some`, the decision at
1140 /// `fork.decision_seq` is overridden (validated against its recomputed
1141 /// legal set) and the acting policy swaps afterward.
1142 fn run_inner(
1143 &self,
1144 config: &TwinRunConfig,
1145 fork: Option<&CounterfactualFork>,
1146 ) -> Result<TwinOutcome> {
1147 if config.workers.is_empty() {
1148 return Err(twin_error(
1149 "the worker pool is empty; placement needs at least one equivalent worker",
1150 ));
1151 }
1152 if config
1153 .timeout_ladder
1154 .iter()
1155 .any(|rung| *rung < MIN_TIMEOUT_RUNG_MS)
1156 {
1157 return Err(twin_error(format!(
1158 "timeout ladder carries a rung below the {MIN_TIMEOUT_RUNG_MS} ms minimum — \
1159 below it ordinary work aborts early, a correctness hazard no policy may cross"
1160 )));
1161 }
1162 let run_id = self.twin_run_id(config.seed);
1163 let mut state = RunState {
1164 journal: Journal::new(
1165 run_id.clone(),
1166 self.thread_id.clone(),
1167 Clock::logical(config.clock.start_ms, config.clock.tick_ms),
1168 ),
1169 rng: RngSource::seeded(config.seed),
1170 run_id,
1171 decisions: Vec::new(),
1172 pending: Vec::new(),
1173 next_decision_seq: 0,
1174 faults_fired: 0,
1175 divergences: Vec::new(),
1176 excluded: Vec::new(),
1177 forked: false,
1178 };
1179 // The acting policy is swappable at exactly one point: the fork.
1180 let mut acting = config.acting.clone();
1181 // Simulated worker lanes: free-at times in milliseconds since run
1182 // start. Steps are barriers; a step starts when every lane is free.
1183 let mut lane_free = vec![0u64; config.workers.len()];
1184 let mut item_elapsed: Vec<u64> = Vec::new();
1185 let mut metrics = TwinMetrics {
1186 items: self.world.items.len(),
1187 ..TwinMetrics::default()
1188 };
1189
1190 for &step in self.world.steps() {
1191 let items = self.world.items_in(step);
1192 if items.is_empty() {
1193 continue;
1194 }
1195 let step_start_time = lane_free.iter().max().copied().unwrap_or(0);
1196 let step_event = state.journal.record(
1197 EventDraft::new(RunEventKind::SuperStepStart, Effect::Pure).input(
1198 serde_json::json!({
1199 "step": step,
1200 "active_nodes": items
1201 .iter()
1202 .map(|item| item.node_id.clone().unwrap_or_else(|| format!("seq:{}", item.recorded_seq)))
1203 .collect::<Vec<_>>(),
1204 }),
1205 ),
1206 );
1207
1208 // The concurrency decision for this step: how many admissions
1209 // the policy allows at once, bounded above by the pool (a
1210 // learned limit may only narrow what the pool admits).
1211 let mut features = Map::new();
1212 features.insert("step".to_owned(), Value::from(step));
1213 features.insert("queue_depth".to_owned(), Value::from(items.len()));
1214 features.insert("workers".to_owned(), Value::from(config.workers.len()));
1215 let legal: Vec<DecisionAction> = config
1216 .concurrency_ladder
1217 .iter()
1218 .map(|limit| DecisionAction::SetConcurrency { limit: *limit })
1219 .collect();
1220 let action = decide(
1221 &mut state,
1222 fork,
1223 &mut acting,
1224 config,
1225 DecisionFamily::Concurrency,
1226 features,
1227 legal,
1228 &step_event,
1229 )?;
1230 let admitted = match action {
1231 DecisionAction::SetConcurrency { limit } => {
1232 (limit as usize).min(config.workers.len()).max(1)
1233 }
1234 _ => 1,
1235 };
1236 flush_pending(&mut state);
1237
1238 // Mechanism 2: the execution order of the parallel task set.
1239 // What the order changes is which item waits behind which on
1240 // the admitted lanes; the journaled order stays canonical
1241 // because each item's drafts are buffered and flushed in
1242 // recorded order after the step executes.
1243 let mut order: Vec<usize> = (0..items.len()).collect();
1244 if config.interleaving == Interleaving::Seeded {
1245 for i in (1..order.len()).rev() {
1246 let j = (state.rng.next_f64() * (i + 1) as f64) as usize;
1247 order.swap(i, j.min(i));
1248 }
1249 }
1250
1251 let mut step_outcomes: BTreeMap<String, &str> = BTreeMap::new();
1252 let mut buffers: Vec<Vec<EventDraft>> = vec![Vec::new(); items.len()];
1253 for &position in &order {
1254 let item = items[position];
1255 let key = format!(
1256 "{}#{}",
1257 item.node_id.as_deref().unwrap_or("seq"),
1258 item.recorded_seq
1259 );
1260 let outcome = run_item(
1261 &mut state,
1262 fork,
1263 &mut acting,
1264 config,
1265 item,
1266 admitted,
1267 &mut lane_free,
1268 &step_event,
1269 step_start_time,
1270 )?;
1271 buffers[position] = std::mem::take(&mut state.pending);
1272 metrics.attempts += outcome.attempts;
1273 metrics.cost_usd += outcome.cost_usd;
1274 item_elapsed.push(outcome.elapsed_ms);
1275 let disposition = match outcome.terminal() {
1276 Terminal::Completed => {
1277 metrics.completed += 1;
1278 "completed"
1279 }
1280 Terminal::DeadLettered => {
1281 metrics.dead_lettered += 1;
1282 "dead_lettered"
1283 }
1284 Terminal::Failed => {
1285 metrics.failed += 1;
1286 "failed"
1287 }
1288 };
1289 step_outcomes.insert(key, disposition);
1290 }
1291
1292 // Canonical flush: the journal's total order of evidence is
1293 // stable across interleavings — execution order only moved the
1294 // timing attributes (admission waits, latencies, timestamps).
1295 for buffer in &mut buffers {
1296 for draft in buffer.drain(..) {
1297 state.journal.record(draft);
1298 }
1299 }
1300
1301 // The step's end evidence is the per-item dispositions — the
1302 // twin's analogue of the recorded journal's channel values, and
1303 // what BranchDiff's step diffs read.
1304 state.journal.record(
1305 EventDraft::new(RunEventKind::SuperStepEnd, Effect::Pure)
1306 .output(serde_json::to_value(&step_outcomes)?)
1307 .parent(step_event),
1308 );
1309 }
1310
1311 metrics.wall_time_ms = lane_free.iter().max().copied().unwrap_or(0);
1312 metrics.item_latency_p50_ms = percentile(&item_elapsed, 50.0);
1313 metrics.item_latency_p95_ms = percentile(&item_elapsed, 95.0);
1314
1315 if let Some(fork) = fork {
1316 if !state.forked {
1317 let case = UnevaluableCase::UnknownDecision {
1318 decision_seq: fork.decision_seq,
1319 };
1320 return Err(twin_error(format!(
1321 "counterfactual refused: the run never reaches decision d{} — {}",
1322 fork.decision_seq,
1323 serde_json::to_string(&case).unwrap_or_else(|_| format!("{case:?}"))
1324 )));
1325 }
1326 }
1327
1328 let journal = state.journal.snapshot();
1329 let report = TwinReport {
1330 run_id: state.run_id.clone(),
1331 seed: config.seed,
1332 faults_declared: config.faults.injections.len(),
1333 faults_fired: state.faults_fired,
1334 decisions: state.next_decision_seq as usize,
1335 evaluable_decisions: state.next_decision_seq as usize,
1336 shadow_divergences: state.divergences,
1337 excluded: state.excluded,
1338 bound: TWIN_REPORT_BOUND.to_owned(),
1339 };
1340 Ok(TwinOutcome {
1341 journal,
1342 decisions: state.decisions,
1343 metrics,
1344 report,
1345 })
1346 }
1347}
1348
1349/// How a work item ended.
1350#[derive(Debug, Clone, Copy, PartialEq, Eq)]
1351enum Terminal {
1352 Completed,
1353 DeadLettered,
1354 Failed,
1355}
1356
1357/// What one work item cost the run.
1358#[derive(Debug, Default)]
1359struct ItemOutcome {
1360 terminal: Option<Terminal>,
1361 attempts: u64,
1362 elapsed_ms: u64,
1363 cost_usd: f64,
1364}
1365
1366impl ItemOutcome {
1367 fn terminal(&self) -> Terminal {
1368 // Every path out of `run_item` sets the terminal state; defaulting
1369 // to `Failed` here would hide a machinery bug as evidence.
1370 self.terminal.expect("run_item always settles its item")
1371 }
1372}
1373
1374/// Mutable per-run evidence state: the journal being written, the decision
1375/// stream, the current item's buffered drafts, and the counters the report
1376/// is built from.
1377struct RunState {
1378 journal: Journal,
1379 rng: RngSource,
1380 run_id: String,
1381 decisions: Vec<DecisionEvent>,
1382 /// Drafts of the item currently executing, flushed in canonical order
1383 /// at the step boundary (see `run_inner`); keeps the journaled total
1384 /// order stable under seeded interleavings.
1385 pending: Vec<EventDraft>,
1386 next_decision_seq: u64,
1387 faults_fired: usize,
1388 divergences: Vec<u64>,
1389 excluded: Vec<UnevaluableCase>,
1390 /// `true` once the fork decision has been taken (a fork that never
1391 /// fires is the `UnknownDecision` refusal).
1392 forked: bool,
1393}
1394
1395/// Write every buffered draft to the journal in order.
1396fn flush_pending(state: &mut RunState) {
1397 for draft in std::mem::take(&mut state.pending) {
1398 state.journal.record(draft);
1399 }
1400}
1401
1402/// One decision point: assemble the evidence, let the acting policy (and
1403/// the shadow) score the same features, validate both against the contract,
1404/// journal both, and return the action that executes.
1405///
1406/// Validation is the policy plane's enforcement at the twin boundary: a
1407/// selection outside `legal_actions` or a propensity outside `(0, 1]` is a
1408/// contract violation, rejected rather than journaled — dishonest evidence
1409/// is worse than no evidence.
1410#[allow(clippy::too_many_arguments)]
1411fn decide(
1412 state: &mut RunState,
1413 fork: Option<&CounterfactualFork>,
1414 acting: &mut Arc<dyn TwinPolicy>,
1415 config: &TwinRunConfig,
1416 family: DecisionFamily,
1417 features: Map<String, Value>,
1418 legal_actions: Vec<DecisionAction>,
1419 parent: &str,
1420) -> Result<DecisionAction> {
1421 let seq = state.next_decision_seq;
1422 let ctx = DecisionContext {
1423 family,
1424 features: &features,
1425 legal_actions: &legal_actions,
1426 };
1427
1428 // Mechanism 3: the fork overrides exactly one decision. The override
1429 // must be legal *here* — the legal set is recomputed from the same
1430 // gates, so an input-changing or gate-crossing action is refused
1431 // against evidence, not against the experimenter's say-so.
1432 let at_fork = fork.is_some_and(|fork| fork.decision_seq == seq);
1433 let (selected, propensity, version) = if at_fork {
1434 let fork = fork.expect("checked above");
1435 if !legal_actions.contains(&fork.action) {
1436 let case = UnevaluableCase::IllegalForkAction {
1437 decision_seq: seq,
1438 action: fork.action.clone(),
1439 };
1440 return Err(twin_error(format!(
1441 "counterfactual refused: {:?} is not in the legal set at decision d{seq} \
1442 ({legal_actions:?}) — {}",
1443 fork.action,
1444 serde_json::to_string(&case).unwrap_or_else(|_| format!("{case:?}"))
1445 )));
1446 }
1447 state.forked = true;
1448 (
1449 fork.action.clone(),
1450 1.0,
1451 PolicyVersion::new(TWIN_FORK_POLICY_VERSION),
1452 )
1453 } else {
1454 let (selected, propensity) = acting.decide(&ctx, state.rng.next_f64());
1455 validate_decision(&selected, propensity, &legal_actions, &acting.version())?;
1456 (selected, propensity, acting.version())
1457 };
1458
1459 let decided_at = state.journal.clock().now();
1460 let acting_event = DecisionEvent {
1461 id: format!("{}:d{seq}", state.run_id),
1462 run_id: state.run_id.clone(),
1463 thread_id: state.journal.thread_id().to_owned(),
1464 seq,
1465 family,
1466 features: features.clone(),
1467 legal_actions: legal_actions.clone(),
1468 selected: selected.clone(),
1469 propensity,
1470 policy_version: version,
1471 role: Some(DecisionRole::Acting),
1472 outcome: None,
1473 decided_at,
1474 };
1475 journal_decision(state, &acting_event, parent);
1476
1477 // Mechanism 4: the shadow scores the same features and journals its
1478 // decision with its true propensity; it never acts. Divergence from
1479 // the acting decision is recorded — those are the points a
1480 // counterfactual branch estimates the candidate's outcome from.
1481 if let Some(shadow) = &config.shadow {
1482 let (shadow_selected, shadow_propensity) = shadow.decide(&ctx, state.rng.next_f64());
1483 validate_decision(
1484 &shadow_selected,
1485 shadow_propensity,
1486 &legal_actions,
1487 &shadow.version(),
1488 )?;
1489 let shadow_event = DecisionEvent {
1490 id: format!("{}:d{seq}:shadow", state.run_id),
1491 selected: shadow_selected.clone(),
1492 propensity: shadow_propensity,
1493 policy_version: shadow.version(),
1494 role: Some(DecisionRole::Shadow),
1495 decided_at: state.journal.clock().now(),
1496 ..acting_event.clone()
1497 };
1498 journal_decision(state, &shadow_event, parent);
1499 if shadow_selected != selected {
1500 state.divergences.push(seq);
1501 }
1502 }
1503
1504 state.next_decision_seq += 1;
1505
1506 // Hybrid replay: after the fork decision, the new policy acts.
1507 if at_fork {
1508 if let Some(next) = fork.and_then(|fork| fork.then_act_with.as_ref()) {
1509 *acting = next.clone();
1510 }
1511 }
1512 Ok(selected)
1513}
1514
1515/// The policy-plane contract, enforced where decisions are journaled.
1516fn validate_decision(
1517 selected: &DecisionAction,
1518 propensity: f64,
1519 legal_actions: &[DecisionAction],
1520 version: &PolicyVersion,
1521) -> Result<()> {
1522 if !legal_actions.contains(selected) {
1523 return Err(twin_error(format!(
1524 "policy `{}` selected {selected:?}, which is outside the legal set \
1525 {legal_actions:?} — a policy chooses among declared members, never free-form \
1526 outputs",
1527 version.as_str()
1528 )));
1529 }
1530 if !(0.0 < propensity && propensity <= 1.0) {
1531 return Err(twin_error(format!(
1532 "policy `{}` declared propensity {propensity}, outside (0, 1] — propensity is \
1533 assigned at decision time and must be truthful, or off-policy evaluation over \
1534 this evidence is meaningless",
1535 version.as_str()
1536 )));
1537 }
1538 Ok(())
1539}
1540
1541/// Buffer one decision as a `PolicyDecision` draft (flushed in canonical
1542/// order at the step boundary) and keep it in the run's decision stream.
1543fn journal_decision(state: &mut RunState, event: &DecisionEvent, parent: &str) {
1544 state.pending.push(
1545 EventDraft::new(RunEventKind::PolicyDecision, Effect::Pure)
1546 .output(
1547 serde_json::to_value(event)
1548 .unwrap_or_else(|_| serde_json::json!({"id": event.id, "error": "decision payload failed to serialize"})),
1549 )
1550 .parent(parent),
1551 );
1552 state.decisions.push(event.clone());
1553}
1554
1555/// What one attempt was observed to do — after faults, timeout bounds, and
1556/// the recorded world's answer are composed. What the policy observes is
1557/// not always what the world did: a bound truncates the observation, the
1558/// truth stays.
1559struct AttemptObservation {
1560 status: EventStatus,
1561 class: Option<ErrorClass>,
1562 retry_after_ms: Option<u64>,
1563 /// Observed attempt latency, excluding admission wait.
1564 latency_ms: u64,
1565 output: Option<Value>,
1566 tokens: Option<Usage>,
1567 cost_usd: Option<f64>,
1568}
1569
1570/// Drive one work item to its terminal state: attempts, faults, decision
1571/// points, and the retry loop, against the simulated worker lanes.
1572#[allow(clippy::too_many_arguments)]
1573fn run_item(
1574 state: &mut RunState,
1575 fork: Option<&CounterfactualFork>,
1576 acting: &mut Arc<dyn TwinPolicy>,
1577 config: &TwinRunConfig,
1578 item: &TwinWorkItem,
1579 admitted: usize,
1580 lane_free: &mut [u64],
1581 step_event: &str,
1582 step_start_time: u64,
1583) -> Result<ItemOutcome> {
1584 let mut outcome = ItemOutcome::default();
1585 let item_ready = step_start_time;
1586 let mut prev_attempt_end = item_ready;
1587 let mut attempt = 0u32;
1588
1589 loop {
1590 attempt += 1;
1591 outcome.attempts += 1;
1592
1593 // Placement: which equivalent worker this attempt runs on. Decided
1594 // per attempt — a retry may re-place, which is exactly what the
1595 // family would learn.
1596 let mut features = Map::new();
1597 features.insert("attempt".to_owned(), Value::from(attempt));
1598 features.insert("effect".to_owned(), serde_json::to_value(item.effect)?);
1599 features.insert(
1600 "lane_free_ms".to_owned(),
1601 serde_json::to_value(&*lane_free)?,
1602 );
1603 if let Some(node) = &item.node_id {
1604 features.insert("node".to_owned(), Value::from(node.clone()));
1605 }
1606 let legal: Vec<DecisionAction> = config
1607 .workers
1608 .iter()
1609 .map(|worker| DecisionAction::SelectWorker {
1610 worker: worker.clone(),
1611 })
1612 .collect();
1613 let action = decide(
1614 state,
1615 fork,
1616 acting,
1617 config,
1618 DecisionFamily::WorkerPlacement,
1619 features,
1620 legal,
1621 step_event,
1622 )?;
1623 let worker = match &action {
1624 DecisionAction::SelectWorker { worker } => worker.clone(),
1625 _ => config.workers[0].clone(),
1626 };
1627 let lane = config
1628 .workers
1629 .iter()
1630 .position(|w| w == &worker)
1631 .unwrap_or(0)
1632 // The concurrency cap admits only so many lanes at once; a
1633 // placement onto a lane beyond the cap queues behind it.
1634 .min(admitted.saturating_sub(1))
1635 .min(lane_free.len() - 1);
1636
1637 // Timeout: the bound this attempt runs under.
1638 let mut features = Map::new();
1639 features.insert(
1640 "recorded_latency_ms".to_owned(),
1641 Value::from(item.answer.latency_ms),
1642 );
1643 features.insert("attempt".to_owned(), Value::from(attempt));
1644 features.insert("effect".to_owned(), serde_json::to_value(item.effect)?);
1645 if let Some(node) = &item.node_id {
1646 features.insert("node".to_owned(), Value::from(node.clone()));
1647 }
1648 let legal: Vec<DecisionAction> = config
1649 .timeout_ladder
1650 .iter()
1651 .map(|millis| DecisionAction::SetTimeout { millis: *millis })
1652 .collect();
1653 let action = decide(
1654 state,
1655 fork,
1656 acting,
1657 config,
1658 DecisionFamily::Timeout,
1659 features,
1660 legal,
1661 step_event,
1662 )?;
1663 let bound = match action {
1664 DecisionAction::SetTimeout { millis } => millis,
1665 _ => MAX_RETRY_DELAY_MS,
1666 };
1667
1668 // The effect boundary: the fault schedule is consulted the way the
1669 // production scheduler reads the world — first matching anchor
1670 // wins, and a fired fault replaces the recorded answer.
1671 let fault = config
1672 .faults
1673 .fault_for_attempt(item.recorded_seq, attempt, &worker)
1674 .cloned();
1675 let mut observation = match &fault {
1676 Some(fault) => {
1677 state.faults_fired += 1;
1678 faulted_observation(fault, bound, item)
1679 }
1680 None => recorded_observation(item, bound),
1681 };
1682
1683 // A crash at the decision point: the scheduler's fail-task path
1684 // observes the in-flight attempt through the injected fault (lease
1685 // expiry classifies `Unknown`). Consulted only where the production
1686 // scheduler would see a lost attempt — a retry decision about to be
1687 // made.
1688 if observation.status == EventStatus::Error {
1689 if let Some(at_decision) = config
1690 .faults
1691 .fault_for_decision(state.next_decision_seq)
1692 .cloned()
1693 {
1694 state.faults_fired += 1;
1695 observation = faulted_observation(&at_decision, bound, item);
1696 }
1697 }
1698
1699 // Admission on the lane: the attempt starts when the lane frees and
1700 // any retry delay has elapsed; the journaled latency is what the
1701 // run experiences — the wait plus the observed attempt.
1702 let ready = if attempt == 1 {
1703 item_ready
1704 } else {
1705 prev_attempt_end
1706 };
1707 let start = ready.max(lane_free[lane]);
1708 let wait = start.saturating_sub(ready);
1709 let end = start + observation.latency_ms;
1710 lane_free[lane] = end;
1711 let journaled_latency = wait + observation.latency_ms;
1712
1713 // Buffer the attempt as the recorded kind with the observed
1714 // outcome (flushed in canonical order at the step boundary).
1715 // Retried attempts re-issue the same bytes — the twin never
1716 // rewrites an effect's input, which is the honest edge made
1717 // structural.
1718 let mut draft = EventDraft::new(item.kind, item.effect)
1719 .input(item.request.clone())
1720 .latency_ms(journaled_latency)
1721 .status(observation.status)
1722 .parent(step_event);
1723 if let Some(node) = &item.node_id {
1724 draft = draft.node(node.clone());
1725 }
1726 match observation.status {
1727 EventStatus::Ok => {
1728 if let Some(output) = &observation.output {
1729 draft = draft.output(output.clone());
1730 }
1731 if let Some(tokens) = observation.tokens {
1732 draft = draft.tokens(tokens);
1733 }
1734 }
1735 EventStatus::Error => {
1736 let class = observation.class.unwrap_or(ErrorClass::Unknown);
1737 draft = draft.output(serde_json::json!({
1738 "error": serde_json::to_value(class)?,
1739 }));
1740 }
1741 EventStatus::Interrupted => unreachable!("servable calls never interrupt"),
1742 }
1743 if let Some(cost) = observation.cost_usd {
1744 draft = draft.cost_usd(cost);
1745 outcome.cost_usd += cost;
1746 }
1747 state.pending.push(draft);
1748 prev_attempt_end = end;
1749
1750 if observation.status == EventStatus::Ok {
1751 outcome.terminal = Some(Terminal::Completed);
1752 outcome.elapsed_ms = end.saturating_sub(item_ready);
1753 return Ok(outcome);
1754 }
1755
1756 // The retry decision: the scheduler's fail-task path. The legal set
1757 // comes from the runtime's own gates — effect gate, class gate,
1758 // attempt budget — so no policy, learned or otherwise, can route
1759 // around them.
1760 let class = observation.class.unwrap_or(ErrorClass::Unknown);
1761 if fault.is_some() && !item.effect.is_freely_repeatable() {
1762 // The fault's future is the gate's, not the policy's: flag it
1763 // so the report names what bounded the evaluation.
1764 let flag = UnevaluableCase::GatedEffect {
1765 effect_seq: item.recorded_seq,
1766 };
1767 if !state.excluded.contains(&flag) {
1768 state.excluded.push(flag);
1769 }
1770 }
1771 let mut features = Map::new();
1772 features.insert("failure_class".to_owned(), serde_json::to_value(class)?);
1773 features.insert("attempt".to_owned(), Value::from(attempt));
1774 features.insert("max_attempts".to_owned(), Value::from(config.max_attempts));
1775 features.insert("effect".to_owned(), serde_json::to_value(item.effect)?);
1776 features.insert(
1777 "dependency_latency_ms".to_owned(),
1778 Value::from(journaled_latency),
1779 );
1780 if let Some(retry_after) = observation.retry_after_ms {
1781 features.insert("retry_after_ms".to_owned(), Value::from(retry_after));
1782 }
1783 let legal = retry_legal_actions(item.effect, class, attempt, config.max_attempts);
1784 // Dead-letter is the retryable-budget-exhausted case: the legal set
1785 // collapses to Abort while the gates would still have allowed a
1786 // retry had budget remained.
1787 let budget_exhausted = item.effect.is_freely_repeatable()
1788 && class.is_retryable()
1789 && attempt >= config.max_attempts;
1790 let action = decide(
1791 state,
1792 fork,
1793 acting,
1794 config,
1795 DecisionFamily::Retry,
1796 features,
1797 legal,
1798 step_event,
1799 )?;
1800 match action {
1801 DecisionAction::Retry { .. } => {
1802 // The delay is policy-parameterized within declared bounds,
1803 // floored by the callee's Retry-After — the one delay the
1804 // world imposes.
1805 let delay = acting
1806 .retry_delay_ms_for(Some(class), attempt, state.rng.next_f64())
1807 .max(observation.retry_after_ms.unwrap_or(0));
1808 lane_free[lane] = lane_free[lane].max(prev_attempt_end + delay);
1809 prev_attempt_end += delay;
1810 }
1811 _ => {
1812 outcome.terminal = Some(if budget_exhausted {
1813 Terminal::DeadLettered
1814 } else {
1815 Terminal::Failed
1816 });
1817 outcome.elapsed_ms = end.saturating_sub(item_ready);
1818 return Ok(outcome);
1819 }
1820 }
1821 }
1822}
1823
1824/// The recorded world's answer, observed through the timeout bound: a
1825/// completing attempt slower than its bound is observed as `Timeout` at the
1826/// bound — the policy's observation changes, the world's truth does not.
1827fn recorded_observation(item: &TwinWorkItem, bound: u64) -> AttemptObservation {
1828 match item.answer.status {
1829 EventStatus::Ok if item.answer.latency_ms > bound => AttemptObservation {
1830 status: EventStatus::Error,
1831 class: Some(ErrorClass::Timeout),
1832 retry_after_ms: None,
1833 latency_ms: bound,
1834 output: None,
1835 tokens: None,
1836 cost_usd: item.answer.cost_usd,
1837 },
1838 _ => AttemptObservation {
1839 status: item.answer.status,
1840 class: (item.answer.status == EventStatus::Error).then_some(ErrorClass::Unknown),
1841 retry_after_ms: None,
1842 latency_ms: item.answer.latency_ms,
1843 output: item.answer.output.clone(),
1844 tokens: item.answer.tokens,
1845 cost_usd: item.answer.cost_usd,
1846 },
1847 }
1848}
1849
1850/// The injected fault as an observation. Latencies follow the world's
1851/// rules, not convenience: a crash is discovered at the lease boundary, a
1852/// hang surfaces at the tighter of the bound and the boundary, and
1853/// fail-fast faults answer at the recorded latency (the attempt ran, then
1854/// the world said no).
1855fn faulted_observation(
1856 fault: &InjectedFault,
1857 bound: u64,
1858 item: &TwinWorkItem,
1859) -> AttemptObservation {
1860 let (latency_ms, retry_after_ms) = match fault {
1861 InjectedFault::WorkerCrash => (MAX_RETRY_DELAY_MS, None),
1862 InjectedFault::CalleeTimeout => (bound.min(MAX_RETRY_DELAY_MS), None),
1863 InjectedFault::RateLimited { retry_after_ms } => {
1864 (item.answer.latency_ms, Some(*retry_after_ms))
1865 }
1866 InjectedFault::ResourceExhausted => (item.answer.latency_ms, None),
1867 };
1868 AttemptObservation {
1869 status: EventStatus::Error,
1870 class: Some(fault.error_class()),
1871 retry_after_ms,
1872 latency_ms,
1873 output: None,
1874 tokens: None,
1875 // Unpriced by the recording, so unpriced here: the attempt count
1876 // is the honest cost proxy, the same accounting Wave 1's
1877 // durable-work class uses.
1878 cost_usd: None,
1879 }
1880}
1881
1882/// Nearest-rank percentile of an unsorted sample set. `pub(crate)`: the
1883/// drift detector (`crate::learn`) reads the same statistic off journaled
1884/// decision features — one nearest-rank definition, shared.
1885pub(crate) fn percentile(samples: &[u64], p: f64) -> u64 {
1886 if samples.is_empty() {
1887 return 0;
1888 }
1889 let mut sorted = samples.to_vec();
1890 sorted.sort_unstable();
1891 let rank = ((p / 100.0) * sorted.len() as f64).ceil() as usize;
1892 sorted[rank.max(1).min(sorted.len()) - 1]
1893}
1894
1895#[cfg(test)]
1896mod tests {
1897 use super::*;
1898 use serde_json::json;
1899
1900 fn ctx<'a>(
1901 family: DecisionFamily,
1902 features: &'a Map<String, Value>,
1903 legal: &'a [DecisionAction],
1904 ) -> DecisionContext<'a> {
1905 DecisionContext {
1906 family,
1907 features,
1908 legal_actions: legal,
1909 }
1910 }
1911
1912 #[test]
1913 fn fault_schedule_matches_in_declaration_order() {
1914 let schedule = FaultSchedule::new(7)
1915 .with_injection(
1916 FaultAnchor::Window {
1917 from_seq: 0,
1918 to_seq: 10,
1919 },
1920 InjectedFault::ResourceExhausted,
1921 )
1922 .with_injection(
1923 FaultAnchor::OnAttempt {
1924 effect_seq: 3,
1925 attempt: 1,
1926 },
1927 InjectedFault::RateLimited {
1928 retry_after_ms: 500,
1929 },
1930 )
1931 .with_injection(
1932 FaultAnchor::OnWorker {
1933 worker: "w1".to_owned(),
1934 },
1935 InjectedFault::WorkerCrash,
1936 );
1937
1938 // The window is declared first and wins for seq 3 despite the more
1939 // specific OnAttempt entry — declaration order is the precedence
1940 // rule, and it is deterministic.
1941 assert_eq!(
1942 schedule.fault_for_attempt(3, 1, "w0"),
1943 Some(&InjectedFault::ResourceExhausted)
1944 );
1945 // Outside the window: the specific anchors answer.
1946 assert_eq!(
1947 schedule.fault_for_attempt(11, 1, "w1"),
1948 Some(&InjectedFault::WorkerCrash)
1949 );
1950 assert_eq!(schedule.fault_for_attempt(11, 1, "w0"), None);
1951 // AtDecision anchors answer only the decision query.
1952 assert_eq!(schedule.fault_for_decision(0), None);
1953 let schedule = schedule.with_injection(
1954 FaultAnchor::AtDecision { decision_seq: 2 },
1955 InjectedFault::WorkerCrash,
1956 );
1957 assert_eq!(
1958 schedule.fault_for_decision(2),
1959 Some(&InjectedFault::WorkerCrash)
1960 );
1961 // Outside the window and off the degraded worker: nothing fires.
1962 assert_eq!(schedule.fault_for_attempt(11, 2, "w0"), None);
1963 }
1964
1965 #[test]
1966 fn injected_faults_classify_into_the_retry_taxonomy() {
1967 assert_eq!(
1968 InjectedFault::WorkerCrash.error_class(),
1969 ErrorClass::Unknown
1970 );
1971 assert_eq!(
1972 InjectedFault::CalleeTimeout.error_class(),
1973 ErrorClass::Timeout
1974 );
1975 assert_eq!(
1976 InjectedFault::RateLimited { retry_after_ms: 50 }.error_class(),
1977 ErrorClass::RateLimited
1978 );
1979 assert_eq!(
1980 InjectedFault::ResourceExhausted.error_class(),
1981 ErrorClass::ResourceExhausted
1982 );
1983 }
1984
1985 #[test]
1986 fn floor_decides_the_static_v0_stance_per_family() {
1987 let features = Map::new();
1988 let floor = StaticFloor;
1989
1990 // Retry: retries while the gates leave a Retry member legal.
1991 let legal = vec![DecisionAction::Retry { attempt: 2 }, DecisionAction::Abort];
1992 let (action, propensity) =
1993 floor.decide(&ctx(DecisionFamily::Retry, &features, &legal), 0.5);
1994 assert_eq!(action, DecisionAction::Retry { attempt: 2 });
1995 assert_eq!(propensity, 1.0);
1996
1997 // Gates closed: only Abort remains.
1998 let legal = vec![DecisionAction::Abort];
1999 let (action, _) = floor.decide(&ctx(DecisionFamily::Retry, &features, &legal), 0.5);
2000 assert_eq!(action, DecisionAction::Abort);
2001
2002 // Timeout: the top rung — "no bound in force" modeled as the lease
2003 // boundary, matching Wave 1's accounting.
2004 let legal: Vec<DecisionAction> = DEFAULT_TIMEOUT_LADDER
2005 .iter()
2006 .map(|millis| DecisionAction::SetTimeout { millis: *millis })
2007 .collect();
2008 let (action, _) = floor.decide(&ctx(DecisionFamily::Timeout, &features, &legal), 0.5);
2009 assert_eq!(
2010 action,
2011 DecisionAction::SetTimeout {
2012 millis: MAX_RETRY_DELAY_MS
2013 }
2014 );
2015
2016 // Concurrency: uncapped. Placement: the first eligible worker (the
2017 // static-pool stance).
2018 let legal = vec![
2019 DecisionAction::SetConcurrency { limit: 1 },
2020 DecisionAction::SetConcurrency { limit: u32::MAX },
2021 ];
2022 let (action, _) = floor.decide(&ctx(DecisionFamily::Concurrency, &features, &legal), 0.5);
2023 assert_eq!(action, DecisionAction::SetConcurrency { limit: u32::MAX });
2024 let legal = vec![
2025 DecisionAction::SelectWorker {
2026 worker: "w0".to_owned(),
2027 },
2028 DecisionAction::SelectWorker {
2029 worker: "w1".to_owned(),
2030 },
2031 ];
2032 let (action, _) = floor.decide(
2033 &ctx(DecisionFamily::WorkerPlacement, &features, &legal),
2034 0.5,
2035 );
2036 assert_eq!(
2037 action,
2038 DecisionAction::SelectWorker {
2039 worker: "w0".to_owned()
2040 }
2041 );
2042 }
2043
2044 #[test]
2045 fn validate_decision_rejects_illegal_actions_and_bad_propensities() {
2046 let legal = vec![DecisionAction::Abort];
2047 let version = PolicyVersion::new("policy-test");
2048 assert!(validate_decision(&DecisionAction::Abort, 1.0, &legal, &version).is_ok());
2049 let error = validate_decision(&DecisionAction::Retry { attempt: 2 }, 1.0, &legal, &version)
2050 .unwrap_err();
2051 assert!(error.to_string().contains("outside the legal set"));
2052 for bad in [0.0, -0.5, 1.5] {
2053 assert!(validate_decision(&DecisionAction::Abort, bad, &legal, &version).is_err());
2054 }
2055 }
2056
2057 #[test]
2058 fn timeout_bound_truncates_the_observation_not_the_world() {
2059 let item = TwinWorkItem {
2060 recorded_seq: 0,
2061 kind: RunEventKind::ToolCall,
2062 node_id: Some("n".to_owned()),
2063 effect: Effect::Idempotent,
2064 request: json!({"tool": "t", "arguments": {}}),
2065 answer: RecordedAnswer {
2066 status: EventStatus::Ok,
2067 output: Some(json!({"r": 1})),
2068 latency_ms: 10_000,
2069 tokens: None,
2070 cost_usd: Some(0.01),
2071 },
2072 step: 0,
2073 };
2074 // Slower than the bound: observed as Timeout at the bound.
2075 let observed = recorded_observation(&item, 5_000);
2076 assert_eq!(observed.status, EventStatus::Error);
2077 assert_eq!(observed.class, Some(ErrorClass::Timeout));
2078 assert_eq!(observed.latency_ms, 5_000);
2079 // Within the bound: the recorded answer, untouched.
2080 let observed = recorded_observation(&item, 30_000);
2081 assert_eq!(observed.status, EventStatus::Ok);
2082 assert_eq!(observed.latency_ms, 10_000);
2083 assert_eq!(observed.output, Some(json!({"r": 1})));
2084 }
2085
2086 #[test]
2087 fn faulted_observations_follow_the_worlds_latency_rules() {
2088 let item = TwinWorkItem {
2089 recorded_seq: 0,
2090 kind: RunEventKind::ToolCall,
2091 node_id: None,
2092 effect: Effect::Idempotent,
2093 request: json!({}),
2094 answer: RecordedAnswer {
2095 status: EventStatus::Ok,
2096 output: None,
2097 latency_ms: 2_000,
2098 tokens: None,
2099 cost_usd: None,
2100 },
2101 step: 0,
2102 };
2103 // A crash is discovered at the lease boundary.
2104 let crash = faulted_observation(&InjectedFault::WorkerCrash, 5_000, &item);
2105 assert_eq!(crash.latency_ms, MAX_RETRY_DELAY_MS);
2106 // A hang surfaces at the tighter of bound and boundary.
2107 let hang = faulted_observation(&InjectedFault::CalleeTimeout, 5_000, &item);
2108 assert_eq!(hang.latency_ms, 5_000);
2109 let hang = faulted_observation(&InjectedFault::CalleeTimeout, MAX_RETRY_DELAY_MS, &item);
2110 assert_eq!(hang.latency_ms, MAX_RETRY_DELAY_MS);
2111 // Rate limits carry their Retry-After floor.
2112 let limited = faulted_observation(
2113 &InjectedFault::RateLimited {
2114 retry_after_ms: 800,
2115 },
2116 5_000,
2117 &item,
2118 );
2119 assert_eq!(limited.retry_after_ms, Some(800));
2120 assert_eq!(limited.latency_ms, 2_000);
2121 }
2122
2123 #[test]
2124 fn percentile_is_nearest_rank() {
2125 assert_eq!(percentile(&[], 50.0), 0);
2126 assert_eq!(percentile(&[10], 95.0), 10);
2127 assert_eq!(percentile(&[1, 2, 3, 4], 50.0), 2);
2128 assert_eq!(percentile(&[1, 2, 3, 4], 95.0), 4);
2129 }
2130}