Skip to main content

rusty_agent_runtime/
twin.rs

1//! The runtime digital twin (R0.10 wave 2): a deterministic re-execution
2//! environment for recorded runs.
3//!
4//! Exact replay ([`crate::replay`]) answers one question — "does the same
5//! graph reproduce the same evidence?" — and stops there. The twin answers
6//! the four questions plain replay cannot, per `docs/adaptation-design.md`:
7//!
8//! 1. **What happens under faults the recording never saw** — a
9//!    [`FaultSchedule`]: a deterministic, seeded list of faults attached to
10//!    a twin run. Injection lands at decision points and the effect
11//!    boundary, never by patching code: the twin's scheduler reads the
12//!    schedule the way the production scheduler reads the world.
13//! 2. **What happens under different concurrent interleavings** — schedule
14//!    randomization: the parallel task set of each super-step is driven in
15//!    a seeded order, so a concurrency policy is evaluated against the
16//!    interleaving distribution, not the one schedule the recording
17//!    happened to get. The journaled total order of evidence stays
18//!    canonical; per-node latencies vary with the schedule.
19//! 3. **What happens if one decision changes** — a [`CounterfactualFork`]:
20//!    apply one different legal action at one decision, continue with
21//!    effects served from the recording where the decision leaves their
22//!    inputs untouched, and compare with [`BranchDiff`]. With
23//!    [`CounterfactualFork::then_act_with`] this is also R0.5's deferred
24//!    *hybrid* replay: the recorded behavior up to the fork, a new policy
25//!    afterward.
26//! 4. **What a candidate policy *would have* decided on the same evidence**
27//!    — a shadow policy: the candidate decides, the floor acts, both
28//!    journal as `PolicyDecision` events ([`DecisionRole::Shadow`] /
29//!    [`DecisionRole::Acting`]) with their true propensities. Exploration
30//!    by seeded draw makes the shadow a stochastic policy with known
31//!    propensities: well-posed off-policy evidence by construction.
32//!
33//! # What the twin reuses (and does not re-implement)
34//!
35//! - **Determinism seams** ([`Clock`], [`RngSource`]): every timestamp,
36//!   backoff jitter draw, interleaving permutation, and stochastic policy
37//!   draw comes from the run's configured seams, so a twin run reproduces
38//!   exactly from its seed and fixture.
39//! - **The effect journal's servable-kind vocabulary**
40//!   ([`crate::replay::SERVABLE_KINDS`]) and integrity verification
41//!   ([`ExactReplay`]'s boundary checks): the recorded world is loaded and
42//!   verified by the replay machinery, then decomposed into work items.
43//! - **The retry policy gates** ([`crate::durable::classify_retry`]'s
44//!   inputs via [`crate::durable::retry_legal_actions`]): the twin never
45//!   re-implements the gates; the legal set a policy sees is computed by
46//!   the same function the production scheduler uses.
47//! - **[`BranchDiff`]**: counterfactual comparison is journaled evidence,
48//!   not a log line.
49//!
50//! # The honest edge
51//!
52//! The twin is a model of the runtime, not of the world. A counterfactual
53//! decision whose downstream effects are replay-servable — retry counts,
54//! backoff delays, timeout bounds, placement among equivalent workers,
55//! concurrency caps, checkpoint cadence — is exactly evaluable, because
56//! those decisions change *when and whether* effects execute, not what
57//! they return. A decision that would change an effect's *input* is
58//! unevaluable: the journal has no answer to a call the recorded world
59//! never received. The twin enforces the boundary structurally — policies
60//! choose among the closed [`DecisionAction`] set of one decision point,
61//! and no member of that set rewrites an effect's input — and refuses
62//! counterfactual forks outside the recomputed legal set. Every
63//! [`TwinReport`] states the bound in its required [`TwinReport::bound`]
64//! field (design open question 5: the constraint lives in the record, not
65//! in prose framing).
66//!
67//! Two modeling decisions, stated plainly:
68//!
69//! - **The world's baseline answer is the recording.** Each recorded
70//!   effect is a work item whose truth is the journaled outcome. Re-served
71//!   attempts (a retry re-issues the same bytes) receive the recorded
72//!   answer again — the world answers the same bytes the same way. A
73//!   recorded error therefore keeps failing across attempts: the recording
74//!   is all the twin knows.
75//! - **The twin's scheduler is synchronous and simulated.** Work items
76//!   execute on declared worker lanes; admission waits, timeout
77//!   truncations, backoff delays, and lease-expiry discoveries are
78//!   computed, not slept. What a policy *observes* (a timeout bound
79//!   truncates the observed latency; the world's truth is untouched) is
80//!   the Wave 1 rule carried into the twin. The graph executor and
81//!   checkpointing are not driven: the twin's unit of evidence is the
82//!   journal it writes, and checkpoint cadence evaluation stays with the
83//!   R0.5 checkpoint machinery.
84
85use std::collections::BTreeMap;
86use std::fmt;
87use std::sync::Arc;
88
89use serde::{Deserialize, Serialize};
90use serde_json::{Map, Value};
91
92use crate::durable::{
93    backoff_delay_ms, backoff_delay_ms_with, resolve_retry_parameters, resolve_timeout_bound_ms,
94    retry_legal_actions, ErrorClass, MAX_RETRY_DELAY_MS,
95};
96use crate::error::{Result, RustyError};
97use crate::journal::{Clock, EventDraft, Journal, JournalSnapshot, RngSource};
98use crate::llm::Usage;
99use crate::record::{
100    DecisionAction, DecisionEvent, DecisionFamily, DecisionRole, Effect, EventStatus,
101    ExecutorPolicy, PayloadRef, PolicyVersion, RunEventKind,
102};
103use crate::replay::{BranchDiff, ExactReplay, LogicalClockParams, ReplayFixture, SERVABLE_KINDS};
104
105fn twin_error(message: impl Into<String>) -> RustyError {
106    // The twin's failures are replay-family failures: a run that cannot
107    // answer from its evidence must stop, not improvise.
108    RustyError::Replay(format!("twin: {}", message.into()))
109}
110
111/// The validity-boundary statement every [`TwinReport`] carries. Required,
112/// not optional: an auditor reads the twin's constraint from the record
113/// itself (design open question 5).
114pub const TWIN_REPORT_BOUND: &str = "the twin evaluates only decisions that change when/whether \
115     effects execute — retry counts, backoff delays, timeout bounds, placement among equivalent \
116     workers, concurrency caps, checkpoint cadence — never what effects return; a decision that \
117     would change an effect's input is unevaluable and was excluded or refused";
118
119/// The policy version a counterfactual fork journals under. A fork is not a
120/// policy — it is the experimenter's directed what-if, deterministic by
121/// construction — so its decisions are named as such rather than
122/// attributed to the floor or the candidate.
123pub const TWIN_FORK_POLICY_VERSION: &str = "twin-fork";
124
125/// The timeout ladder's minimum rung, in milliseconds. Below this, ordinary
126/// work aborts early — a correctness hazard no policy may cross (the same
127/// floor Wave 1 pre-registered). The constant lives with the family's
128/// parameter contract ([`crate::record::MIN_TIMEOUT_RUNG_MS`]); the twin
129/// re-exports it so the evaluation harness and the production read path
130/// enforce one bound.
131pub use crate::record::MIN_TIMEOUT_RUNG_MS;
132
133/// The default timeout ladder: discrete rungs between the minimum and the
134/// lease boundary ([`MAX_RETRY_DELAY_MS`], the queue's worst-case discovery
135/// latency for hung work — the floor's "no bound in force", modeled as the
136/// top rung exactly as Wave 1 modeled it).
137pub const DEFAULT_TIMEOUT_LADDER: [u64; 7] =
138    [100, 500, 1_000, 5_000, 30_000, 120_000, MAX_RETRY_DELAY_MS];
139
140/// The default concurrency ladder. The top rung is uncapped — the floor's
141/// stance when no concurrency policy is in force.
142pub const DEFAULT_CONCURRENCY_LADDER: [u32; 6] = [1, 2, 4, 8, 16, u32::MAX];
143
144// ---------------------------------------------------------------------------
145// Mechanism 1: fault injection.
146// ---------------------------------------------------------------------------
147
148/// One injectable fault. The taxonomy mirrors the world's failure modes the
149/// scheduler already classifies: each variant names the [`ErrorClass`] the
150/// injected failure is observed with.
151#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
152#[serde(tag = "fault", rename_all = "snake_case")]
153pub enum InjectedFault {
154    /// The worker dies mid-attempt. The attempt is lost and surfaces at the
155    /// lease boundary ([`MAX_RETRY_DELAY_MS`]) classified
156    /// [`ErrorClass::Unknown`] — the lease-expiry path, the dead-letter
157    /// queue's primary input.
158    WorkerCrash,
159
160    /// The attempt never completes on its own. With no timeout bound in
161    /// force it surfaces at the lease boundary; with a bound, at the bound,
162    /// classified [`ErrorClass::Timeout`] either way.
163    CalleeTimeout,
164
165    /// The callee asks to be slowed down: [`ErrorClass::RateLimited`] with
166    /// a `Retry-After` floor any policy's delay must respect.
167    RateLimited {
168        /// The callee-supplied `Retry-After` floor, in milliseconds.
169        retry_after_ms: u64,
170    },
171
172    /// The worker or callee is out of capacity:
173    /// [`ErrorClass::ResourceExhausted`]. Retryable — ideally on a
174    /// different worker, which is what makes this the placement family's
175    /// fault.
176    ResourceExhausted,
177}
178
179impl InjectedFault {
180    /// The class the scheduler observes for this fault.
181    pub fn error_class(&self) -> ErrorClass {
182        match self {
183            InjectedFault::WorkerCrash => ErrorClass::Unknown,
184            InjectedFault::CalleeTimeout => ErrorClass::Timeout,
185            InjectedFault::RateLimited { .. } => ErrorClass::RateLimited,
186            InjectedFault::ResourceExhausted => ErrorClass::ResourceExhausted,
187        }
188    }
189}
190
191/// Where a fault lands. Injection happens at decision points and the effect
192/// boundary — the two places the production scheduler reads the world.
193#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
194#[serde(tag = "anchor", rename_all = "snake_case")]
195pub enum FaultAnchor {
196    /// At the effect boundary: the `attempt`-th attempt (1-based) of the
197    /// effect recorded at `effect_seq` fails with the injected fault.
198    OnAttempt {
199        /// The recorded journal sequence number of the target effect.
200        effect_seq: u64,
201        /// The 1-based attempt ordinal the fault fires on.
202        attempt: u32,
203    },
204
205    /// At a decision point: when the scheduler makes the retry decision
206    /// with sequence `decision_seq`, the in-flight attempt is observed
207    /// through the injected fault instead of its real outcome (a crash
208    /// here is the lease-expiry path: the worker died, the attempt is
209    /// lost, the failure classifies [`ErrorClass::Unknown`]). Fires only
210    /// at retry decision points — the scheduler's fail-task path, where
211    /// the production scheduler would observe a lost attempt; an anchor
212    /// that never coincides with one never fires, and the report's
213    /// fired/declared counts say so.
214    AtDecision {
215        /// The twin run's decision sequence number (`d{n}`) to fire at.
216        decision_seq: u64,
217    },
218
219    /// A provider rate-limit window: every attempt of every effect recorded
220    /// in `from_seq..=to_seq` is answered with the injected fault — the
221    /// provider-side degradation a recording from a healthy window never
222    /// contains.
223    Window {
224        /// First recorded effect sequence covered by the window.
225        from_seq: u64,
226        /// Last recorded effect sequence covered by the window (inclusive).
227        to_seq: u64,
228    },
229
230    /// Resource exhaustion on one worker: every attempt placed on `worker`
231    /// fails. The placement family's evaluation fault: work steered onto
232    /// the degraded worker pays for it.
233    OnWorker {
234        /// The degraded worker's identity (a member of the run's pool).
235        worker: String,
236    },
237}
238
239/// One entry of a [`FaultSchedule`]: an anchor plus the fault that fires
240/// there.
241#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
242pub struct FaultInjection {
243    /// Where the fault lands.
244    #[serde(flatten)]
245    pub anchor: FaultAnchor,
246
247    /// What the world does there.
248    #[serde(flatten)]
249    pub fault: InjectedFault,
250}
251
252/// A deterministic, seeded fault schedule attached to a twin run.
253///
254/// The schedule is *data*, declared before the run: the twin's scheduler
255/// reads it the way the production scheduler reads the world, and two runs
256/// over the same fixture with the same schedule and seed fire exactly the
257/// same faults at exactly the same points. Serializable so a schedule can
258/// be committed alongside the fixture it probes — the evaluation's
259/// reproducibility artifact.
260#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
261pub struct FaultSchedule {
262    /// The seed the schedule was authored under (carried for provenance;
263    /// the run's own seed governs draws).
264    pub seed: u64,
265
266    /// The declared injections, consulted in declaration order: the first
267    /// anchor matching an attempt or decision point wins.
268    #[serde(default, skip_serializing_if = "Vec::is_empty")]
269    pub injections: Vec<FaultInjection>,
270}
271
272impl FaultSchedule {
273    /// An empty schedule under `seed`: the recorded world, unfaulted.
274    pub fn new(seed: u64) -> Self {
275        Self {
276            seed,
277            injections: Vec::new(),
278        }
279    }
280
281    /// Builder-style: declare one more injection.
282    pub fn with_injection(mut self, anchor: FaultAnchor, fault: InjectedFault) -> Self {
283        self.injections.push(FaultInjection { anchor, fault });
284        self
285    }
286
287    /// The fault firing on attempt `attempt` of the effect recorded at
288    /// `effect_seq` when placed on `worker`, if any. Consulted at the
289    /// effect boundary, once per attempt.
290    pub fn fault_for_attempt(
291        &self,
292        effect_seq: u64,
293        attempt: u32,
294        worker: &str,
295    ) -> Option<&InjectedFault> {
296        self.injections
297            .iter()
298            .find(|injection| match &injection.anchor {
299                FaultAnchor::OnAttempt {
300                    effect_seq: seq,
301                    attempt: ord,
302                } => *seq == effect_seq && *ord == attempt,
303                FaultAnchor::Window { from_seq, to_seq } => {
304                    *from_seq <= effect_seq && effect_seq <= *to_seq
305                }
306                FaultAnchor::OnWorker { worker: w } => w == worker,
307                FaultAnchor::AtDecision { .. } => false,
308            })
309            .map(|injection| &injection.fault)
310    }
311
312    /// The fault firing at the retry decision with sequence `decision_seq`,
313    /// if any. Consulted at the scheduler's fail-task decision point only.
314    pub fn fault_for_decision(&self, decision_seq: u64) -> Option<&InjectedFault> {
315        self.injections
316            .iter()
317            .find(|injection| match &injection.anchor {
318                FaultAnchor::AtDecision { decision_seq: seq } => *seq == decision_seq,
319                _ => false,
320            })
321            .map(|injection| &injection.fault)
322    }
323}
324
325// ---------------------------------------------------------------------------
326// Mechanism 4's other half: the policies. (The shadow pair itself is the
327// run machinery below.)
328// ---------------------------------------------------------------------------
329
330/// What a policy sees at one decision point: the family, the journaled
331/// feature map, and the closed legal-action set computed by the runtime's
332/// own gates. A policy chooses among `legal_actions` members and declares
333/// its propensity — never a free-form output.
334#[derive(Debug)]
335pub struct DecisionContext<'a> {
336    /// Which executor decision this is.
337    pub family: DecisionFamily,
338
339    /// The observation assembled at the decision point.
340    pub features: &'a Map<String, Value>,
341
342    /// Every action legal at decision time.
343    pub legal_actions: &'a [DecisionAction],
344}
345
346/// A policy the twin can evaluate: the floor, or a candidate scoring the
347/// same features.
348///
349/// `decide` receives one seeded `draw` per invocation so stochastic
350/// candidates explore by seeded draw — a stochastic policy with known
351/// propensities, which is what makes its journaled shadow decisions
352/// well-posed off-policy evidence. Implementations must return a member of
353/// `legal_actions` and a propensity in `(0, 1]`; the twin rejects either
354/// violation at the boundary rather than journaling dishonest evidence.
355pub trait TwinPolicy: fmt::Debug + Send + Sync {
356    /// The version the policy's decisions journal under.
357    fn version(&self) -> PolicyVersion;
358
359    /// Choose an action from `ctx.legal_actions` and declare its
360    /// propensity. `draw` is a seeded uniform from `[0, 1)`.
361    fn decide(&self, ctx: &DecisionContext<'_>, draw: f64) -> (DecisionAction, f64);
362
363    /// The re-queue delay after a `Retry` decision, in milliseconds. The
364    /// closed action set carries no delay member — the delay is
365    /// policy-parameterized within declared bounds, per the family's
366    /// contract — so it is a method, not an action. Defaults to the
367    /// floor's full-jitter exponential backoff.
368    fn retry_delay_ms(&self, attempt: u32, draw: f64) -> u64 {
369        backoff_delay_ms(attempt, draw)
370    }
371
372    /// The re-queue delay with the failure class in scope (R0.10 wave 3):
373    /// learned policies tune the schedule per class
374    /// ([`ParameterizedPolicy`]); policies that do not override this keep
375    /// [`TwinPolicy::retry_delay_ms`]'s class-blind behavior. The default
376    /// delegates, so every pre-wave-3 policy is unchanged.
377    fn retry_delay_ms_for(&self, class: Option<ErrorClass>, attempt: u32, draw: f64) -> u64 {
378        let _ = class;
379        self.retry_delay_ms(attempt, draw)
380    }
381}
382
383/// The `static-v0` floor: the behavior every pre-learning run had, and the
384/// baseline every candidate is evaluated against.
385///
386/// Per family: **retry** retries whenever the gates leave a `Retry` member
387/// in the legal set (the legal set is computed by
388/// [`retry_legal_actions`], so the floor's stance is exactly
389/// `classify_retry`'s); **timeout** imposes no bound — modeled as the
390/// ladder's top rung, the lease boundary, Wave 1's `LEASE_BOUND_MS`;
391/// **placement** takes the first eligible worker (the static-pool stance:
392/// no re-placement intelligence); **concurrency** is uncapped. Every
393/// selection has propensity 1.0: the floor is deterministic.
394#[derive(Debug, Default, Clone, Copy)]
395pub struct StaticFloor;
396
397impl TwinPolicy for StaticFloor {
398    fn version(&self) -> PolicyVersion {
399        PolicyVersion::new(PolicyVersion::STATIC_V0)
400    }
401
402    fn decide(&self, ctx: &DecisionContext<'_>, _draw: f64) -> (DecisionAction, f64) {
403        let selected = match ctx.family {
404            DecisionFamily::Retry => ctx
405                .legal_actions
406                .iter()
407                .find(|action| matches!(action, DecisionAction::Retry { .. }))
408                .cloned()
409                .unwrap_or(DecisionAction::Abort),
410            // The floor's "no bound in force" and "unlimited" stances are
411            // the ladder tops; placement's static pool is the first
412            // eligible worker; the checkpoint floor writes.
413            DecisionFamily::Timeout
414            | DecisionFamily::Concurrency
415            | DecisionFamily::CheckpointPlacement => ctx
416                .legal_actions
417                .last()
418                .cloned()
419                .unwrap_or(DecisionAction::Abort),
420            DecisionFamily::WorkerPlacement => ctx
421                .legal_actions
422                .first()
423                .cloned()
424                .unwrap_or(DecisionAction::Abort),
425        };
426        (selected, 1.0)
427    }
428}
429
430/// The application loop as a twin policy (R0.10 wave 3): an
431/// [`ExecutorPolicy`] — the registry body a promoted candidate produced —
432/// deciding every family with its parameters, so the twin prices the
433/// exact behavior a bound run would execute.
434///
435/// Per family: **retry** applies the per-class backoff table and narrowed
436/// attempt budget ([`resolve_retry_parameters`]) — the gates stay the
437/// runtime's own (the legal set is computed before the policy sees it, so
438/// no parameter set can route around them); **timeout** applies the
439/// per-callee/default bound ([`resolve_timeout_bound_ms`]) honored by the
440/// smallest ladder rung that covers it; **placement, concurrency, and
441/// checkpoint placement decide exactly as the floor** — those families
442/// have no parameter contract ([`ExecutorPolicy::with_family_parameters`]
443/// rejects them), so a registered policy can never carry their parameters
444/// and there is nothing to apply. Every selection is deterministic
445/// (propensity 1.0): learned exploration belongs to shadow candidates
446/// with declared propensities, not to a promoted body.
447///
448/// Fail-closed at two layers, matching the production read path:
449/// construction refuses an invalid policy ([`ExecutorPolicy::validate`] —
450/// an out-of-envelope body cannot even be evaluated), and the
451/// `resolve_*` functions re-check at every decision point, so a policy
452/// that somehow reaches a decision invalid steers nothing and the floor
453/// decides.
454#[derive(Debug, Clone)]
455pub struct ParameterizedPolicy {
456    policy: ExecutorPolicy,
457    version: PolicyVersion,
458}
459
460impl ParameterizedPolicy {
461    /// A policy deciding with `policy`'s parameters, journaling under
462    /// `version` (the body the registry holds for that version —
463    /// mismatched pairs are the caller's integrity failure to make).
464    pub fn new(policy: ExecutorPolicy, version: PolicyVersion) -> Result<Self> {
465        policy.validate().map_err(|e| {
466            twin_error(format!(
467                "refusing to evaluate policy `{version}`: its parameters fall outside the \
468                 envelope ({e}) — an invalid body cannot act, in evaluation or in production"
469            ))
470        })?;
471        Ok(Self { policy, version })
472    }
473
474    /// The parameters this policy decides with.
475    pub fn policy(&self) -> &ExecutorPolicy {
476        &self.policy
477    }
478
479    /// Read one error class out of a decision's journaled features (the
480    /// retry family's `failure_class` key — absent or unparseable yields
481    /// `Unknown`, the DLQ's conservative default).
482    fn failure_class(ctx: &DecisionContext<'_>) -> ErrorClass {
483        ctx.features
484            .get("failure_class")
485            .and_then(|value| serde_json::from_value(value.clone()).ok())
486            .unwrap_or(ErrorClass::Unknown)
487    }
488
489    /// Read a `u64` feature by key, when present.
490    fn u64_feature(ctx: &DecisionContext<'_>, key: &str) -> Option<u64> {
491        ctx.features.get(key).and_then(serde_json::Value::as_u64)
492    }
493}
494
495impl TwinPolicy for ParameterizedPolicy {
496    fn version(&self) -> PolicyVersion {
497        self.version.clone()
498    }
499
500    fn decide(&self, ctx: &DecisionContext<'_>, _draw: f64) -> (DecisionAction, f64) {
501        let selected = match ctx.family {
502            DecisionFamily::Retry => {
503                let class = Self::failure_class(ctx);
504                let attempt = Self::u64_feature(ctx, "attempt").unwrap_or(1) as u32;
505                let budget = Self::u64_feature(ctx, "max_attempts").unwrap_or(3) as u32;
506                let resolved = resolve_retry_parameters(&self.policy, class, budget);
507                // The policy's budget narrows the decision, never the legal
508                // set: retrying stays legal (the runtime's gates said so),
509                // the policy declines it past its learned budget.
510                if attempt < resolved.max_attempts {
511                    ctx.legal_actions
512                        .iter()
513                        .find(|action| matches!(action, DecisionAction::Retry { .. }))
514                        .cloned()
515                        .unwrap_or(DecisionAction::Abort)
516                } else {
517                    DecisionAction::Abort
518                }
519            }
520            DecisionFamily::Timeout => {
521                // The twin journals the callee identity as `node` (the node
522                // that issued the call); the production emission point
523                // journals `callee`. Both name the same identity.
524                let callee = ctx
525                    .features
526                    .get("callee")
527                    .or_else(|| ctx.features.get("node"))
528                    .and_then(serde_json::Value::as_str);
529                let bound = resolve_timeout_bound_ms(&self.policy, callee);
530                match bound {
531                    Some(bound) => ctx
532                        .legal_actions
533                        .iter()
534                        .find(|action| matches!(action, DecisionAction::SetTimeout { millis } if *millis >= bound))
535                        .cloned()
536                        .or_else(|| ctx.legal_actions.last().cloned())
537                        .unwrap_or(DecisionAction::Abort),
538                    // No bound in force: the floor's stance, the top rung.
539                    None => ctx.legal_actions.last().cloned().unwrap_or(DecisionAction::Abort),
540                }
541            }
542            // Shadow-only families decide exactly as the floor: no
543            // parameter contract exists, so there is nothing to apply.
544            DecisionFamily::Concurrency | DecisionFamily::CheckpointPlacement => ctx
545                .legal_actions
546                .last()
547                .cloned()
548                .unwrap_or(DecisionAction::Abort),
549            DecisionFamily::WorkerPlacement => ctx
550                .legal_actions
551                .first()
552                .cloned()
553                .unwrap_or(DecisionAction::Abort),
554        };
555        (selected, 1.0)
556    }
557
558    fn retry_delay_ms_for(&self, class: Option<ErrorClass>, attempt: u32, draw: f64) -> u64 {
559        let retry = &self.policy.retry;
560        if retry.validate().is_err() {
561            return backoff_delay_ms(attempt, draw);
562        }
563        let schedule = class
564            .and_then(|class| retry.per_class.as_ref()?.get(&class).copied())
565            .unwrap_or_else(|| retry.flat());
566        backoff_delay_ms_with(attempt, draw, schedule.base_delay_ms, schedule.max_delay_ms)
567    }
568}
569
570// ---------------------------------------------------------------------------
571// The recorded world: a journaled run decomposed into work items.
572// ---------------------------------------------------------------------------
573
574/// What the recorded world did for one effect: the journaled outcome,
575/// resolved out of the snapshot's payload references.
576#[derive(Debug, Clone)]
577pub struct RecordedAnswer {
578    /// How the recorded call ended (`Interrupted` is rejected at load:
579    /// servable calls never produce it).
580    pub status: EventStatus,
581
582    /// The recorded response payload.
583    pub output: Option<Value>,
584
585    /// The recorded latency in milliseconds (0 when unmeasured).
586    pub latency_ms: u64,
587
588    /// The recorded token usage, when reported.
589    pub tokens: Option<Usage>,
590
591    /// The recorded cost in USD, when known.
592    pub cost_usd: Option<f64>,
593}
594
595/// One recorded effect as a unit of twin work: the request the world
596/// answered, the answer it gave, the effect classification the gates read,
597/// and the super-step the journal places it in.
598#[derive(Debug, Clone)]
599pub struct TwinWorkItem {
600    /// The recorded event's journal sequence number (fault anchors name
601    /// effects by it).
602    pub recorded_seq: u64,
603
604    /// The recorded effect kind (a member of [`SERVABLE_KINDS`]).
605    pub kind: RunEventKind,
606
607    /// The node that issued the call, when journaled.
608    pub node_id: Option<String>,
609
610    /// The declared effect class — the retry gate's input.
611    pub effect: Effect,
612
613    /// The request, exactly as journaled. Retried attempts re-issue these
614    /// bytes; the twin never rewrites them.
615    pub request: Value,
616
617    /// What the recorded world answered.
618    pub answer: RecordedAnswer,
619
620    /// The super-step the recording placed this effect in (0 when the
621    /// journal carries no super-step structure).
622    pub step: u64,
623}
624
625/// The recorded world a twin run re-executes against: the fixture's
626/// servable effects as work items, grouped by super-step.
627#[derive(Debug, Clone)]
628pub struct TwinWorld {
629    items: Vec<TwinWorkItem>,
630    /// Distinct super-step indexes in ascending order.
631    steps: Vec<u64>,
632}
633
634impl TwinWorld {
635    /// Decompose an integrity-verified snapshot into work items. Effects
636    /// journaled between `SuperStepStart` boundaries inherit that step;
637    /// everything before the first boundary (or in a journal with no
638    /// boundaries) lands in step 0.
639    pub fn from_snapshot(snapshot: &JournalSnapshot) -> Result<Self> {
640        let resolve = |payload: &PayloadRef| -> Option<Value> {
641            match payload {
642                PayloadRef::Inline(value) => Some(value.clone()),
643                PayloadRef::Artifact(reference) => {
644                    snapshot.artifacts.get(&reference.sha256).cloned()
645                }
646            }
647        };
648        let mut items = Vec::new();
649        let mut steps = Vec::new();
650        let mut current_step = 0u64;
651        for event in &snapshot.events {
652            if event.kind == RunEventKind::SuperStepStart {
653                current_step = event
654                    .input
655                    .as_ref()
656                    .and_then(&resolve)
657                    .and_then(|input| input.get("step").and_then(Value::as_u64))
658                    .unwrap_or(current_step + 1);
659                steps.push(current_step);
660                continue;
661            }
662            if !SERVABLE_KINDS.contains(&event.kind) {
663                continue;
664            }
665            if event.status == EventStatus::Interrupted {
666                return Err(twin_error(format!(
667                    "recorded {:?} at seq {} has status `interrupted`, which servable calls \
668                     never produce — the journal is inconsistent",
669                    event.kind, event.seq
670                )));
671            }
672            let request = event.input.as_ref().and_then(&resolve).ok_or_else(|| {
673                twin_error(format!(
674                    "recorded {:?} at seq {} has no input payload; the twin needs the request \
675                     the world answered",
676                    event.kind, event.seq
677                ))
678            })?;
679            let answer = RecordedAnswer {
680                status: event.status,
681                output: event.output.as_ref().and_then(&resolve),
682                latency_ms: event.latency_ms.unwrap_or(0),
683                tokens: event.tokens,
684                cost_usd: event.cost_usd,
685            };
686            items.push(TwinWorkItem {
687                recorded_seq: event.seq,
688                kind: event.kind,
689                node_id: event.node_id.clone(),
690                effect: event.effect,
691                request,
692                answer,
693                step: current_step,
694            });
695        }
696        if steps.is_empty() && !items.is_empty() {
697            steps.push(0);
698        }
699        Ok(Self { items, steps })
700    }
701
702    /// Every work item, in recorded sequence order.
703    pub fn items(&self) -> &[TwinWorkItem] {
704        &self.items
705    }
706
707    /// The super-step indexes the recording contains, ascending.
708    pub fn steps(&self) -> &[u64] {
709        &self.steps
710    }
711
712    /// The work items of one super-step, in recorded (canonical) order.
713    pub fn items_in(&self, step: u64) -> Vec<&TwinWorkItem> {
714        self.items.iter().filter(|item| item.step == step).collect()
715    }
716}
717
718// ---------------------------------------------------------------------------
719// Run configuration and outcomes.
720// ---------------------------------------------------------------------------
721
722/// How the parallel task set of each super-step is ordered.
723#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
724#[serde(rename_all = "snake_case")]
725pub enum Interleaving {
726    /// Recorded order. The baseline every randomized run compares against.
727    #[default]
728    Canonical,
729
730    /// A seeded permutation drawn from the run's [`RngSource`] — mechanism
731    /// 2: the same evidence re-driven under one point of the interleaving
732    /// distribution. Journaled event order stays canonical; admission
733    /// waits, and therefore per-node latencies, follow the drawn order.
734    Seeded,
735}
736
737/// The parameters of one twin run: determinism seams, the fault schedule,
738/// the acting policy, an optional shadow, the decision ladders, and the
739/// worker pool.
740///
741/// Two runs over the same [`Twin`] with equal configs produce byte-identical
742/// journals — the twin's whole claim, proven in the test suite across
743/// repeated runs and across process invocations (the checked-in golden).
744pub struct TwinRunConfig {
745    /// The run's seed: RNG draws, interleaving permutations, and stochastic
746    /// policy draws all derive from it.
747    pub seed: u64,
748
749    /// The logical clock the run's journal timestamps from.
750    pub clock: LogicalClockParams,
751
752    /// The fault schedule the scheduler reads. Empty means the recorded
753    /// world, unfaulted.
754    pub faults: FaultSchedule,
755
756    /// The policy whose actions execute. The floor by default.
757    pub acting: Arc<dyn TwinPolicy>,
758
759    /// A candidate scoring the same features without acting (mechanism 4).
760    pub shadow: Option<Arc<dyn TwinPolicy>>,
761
762    /// The timeout family's legal rungs, ascending. The top rung models
763    /// "no bound in force"; keep it at [`MAX_RETRY_DELAY_MS`] for
764    /// comparability with Wave 1.
765    pub timeout_ladder: Vec<u64>,
766
767    /// The concurrency family's legal rungs, ascending; `u32::MAX` is
768    /// uncapped.
769    pub concurrency_ladder: Vec<u32>,
770
771    /// The worker pool placement ranks. Equivalence is a precondition: the
772    /// pool declares the workers interchangeable for this workload.
773    pub workers: Vec<String>,
774
775    /// The retry attempt budget (the floor's own is 3).
776    pub max_attempts: u32,
777
778    /// Canonical or seeded ordering of each super-step's task set.
779    pub interleaving: Interleaving,
780}
781
782impl TwinRunConfig {
783    /// A baseline configuration: the floor acting, no shadow, no faults,
784    /// canonical order, default ladders, one worker, the floor's attempt
785    /// budget.
786    pub fn new(seed: u64) -> Self {
787        Self {
788            seed,
789            clock: LogicalClockParams {
790                start_ms: 1_700_000_000_000,
791                tick_ms: 10,
792            },
793            faults: FaultSchedule::new(seed),
794            acting: Arc::new(StaticFloor),
795            shadow: None,
796            timeout_ladder: DEFAULT_TIMEOUT_LADDER.to_vec(),
797            concurrency_ladder: DEFAULT_CONCURRENCY_LADDER.to_vec(),
798            workers: vec!["worker-0".to_owned()],
799            max_attempts: 3,
800            interleaving: Interleaving::Canonical,
801        }
802    }
803
804    /// Builder-style: attach a fault schedule.
805    pub fn with_faults(mut self, faults: FaultSchedule) -> Self {
806        self.faults = faults;
807        self
808    }
809
810    /// Builder-style: act with `policy` instead of the floor.
811    pub fn with_acting(mut self, policy: Arc<dyn TwinPolicy>) -> Self {
812        self.acting = policy;
813        self
814    }
815
816    /// Builder-style: shadow the acting policy with `candidate`.
817    pub fn with_shadow(mut self, candidate: Arc<dyn TwinPolicy>) -> Self {
818        self.shadow = Some(candidate);
819        self
820    }
821
822    /// Builder-style: declare the worker pool.
823    pub fn with_workers(mut self, workers: impl IntoIterator<Item = impl Into<String>>) -> Self {
824        self.workers = workers.into_iter().map(Into::into).collect();
825        self
826    }
827
828    /// Builder-style: override the retry attempt budget.
829    pub fn with_max_attempts(mut self, max_attempts: u32) -> Self {
830        self.max_attempts = max_attempts;
831        self
832    }
833
834    /// Builder-style: order each super-step's task set by seeded draw.
835    pub fn with_seeded_interleaving(mut self) -> Self {
836        self.interleaving = Interleaving::Seeded;
837        self
838    }
839}
840
841impl fmt::Debug for TwinRunConfig {
842    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
843        f.debug_struct("TwinRunConfig")
844            .field("seed", &self.seed)
845            .field("clock", &self.clock)
846            .field("faults", &self.faults)
847            .field("acting", &self.acting.version())
848            .field("shadow", &self.shadow.as_ref().map(|p| p.version()))
849            .field("workers", &self.workers)
850            .field("max_attempts", &self.max_attempts)
851            .field("interleaving", &self.interleaving)
852            .finish()
853    }
854}
855
856/// Aggregate outcome metrics of one twin run. Simulated, deterministic, and
857/// directly comparable across arms — the same accounting Wave 1 prices.
858#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
859pub struct TwinMetrics {
860    /// Work items the run contained.
861    pub items: usize,
862
863    /// Items that reached a successful terminal attempt.
864    pub completed: usize,
865
866    /// Items that exhausted the attempt budget on a retryable failure.
867    pub dead_lettered: usize,
868
869    /// Items that failed permanently (the effect or class gate).
870    pub failed: usize,
871
872    /// Total attempts executed across all items.
873    pub attempts: u64,
874
875    /// Summed recorded cost in USD across served attempts.
876    pub cost_usd: f64,
877
878    /// Simulated wall time of the whole run, in milliseconds.
879    pub wall_time_ms: u64,
880
881    /// Nearest-rank p50 of per-item elapsed time (first ready to terminal),
882    /// in milliseconds.
883    pub item_latency_p50_ms: u64,
884
885    /// Nearest-rank p95 of per-item elapsed time, in milliseconds.
886    pub item_latency_p95_ms: u64,
887}
888
889/// A case the twin excluded or flagged under the honest edge. Structural
890/// prevention covers most of the boundary (no legal action rewrites an
891/// effect's input); these are the cases that still need a name in the
892/// report.
893#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
894#[serde(tag = "case", rename_all = "snake_case")]
895pub enum UnevaluableCase {
896    /// A fault fired on work whose declared effect is not freely
897    /// repeatable: the effect gate fails the item on the first fault, so
898    /// retry counterfactuals past that point have no reachable world.
899    /// Flagged rather than silent: a schedule author should read that the
900    /// gate, not the policy, decided the item's future.
901    GatedEffect {
902        /// The recorded sequence number of the gated effect.
903        effect_seq: u64,
904    },
905
906    /// A counterfactual fork named an action outside the legal set
907    /// recomputed at its decision point — including every input-changing
908    /// action, which no legal set ever contains. Always a refusal, never a
909    /// journaled run.
910    IllegalForkAction {
911        /// The decision the fork named.
912        decision_seq: u64,
913        /// The action that was refused.
914        action: DecisionAction,
915    },
916
917    /// A counterfactual fork named a decision the run never reaches.
918    UnknownDecision {
919        /// The decision sequence that does not exist in this run.
920        decision_seq: u64,
921    },
922}
923
924/// The report every twin run emits: what was evaluated, what was excluded,
925/// and the bound the evaluation holds to. The validity boundary is a
926/// required field of the payload, not prose about it.
927#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
928pub struct TwinReport {
929    /// The twin run's id.
930    pub run_id: String,
931
932    /// The seed the run drew from.
933    pub seed: u64,
934
935    /// Faults the schedule declared.
936    pub faults_declared: usize,
937
938    /// Faults that actually fired. Less than declared means anchors that
939    /// never coincided with an attempt or retry decision — visible, never
940    /// silent.
941    pub faults_fired: usize,
942
943    /// Decisions journaled by the acting policy (shadow decisions add
944    /// their own events but are not separate decision points).
945    pub decisions: usize,
946
947    /// Decision points whose outcomes are exactly evaluable: replay-servable
948    /// or fault-injected downstream. In this model every journaled decision
949    /// is evaluable by construction; the field exists so a future mechanism
950    /// that is not must subtract here.
951    pub evaluable_decisions: usize,
952
953    /// Decision sequences where the shadow's selection differed from the
954    /// acting policy's — the points a counterfactual branch estimates the
955    /// candidate's outcome from.
956    pub shadow_divergences: Vec<u64>,
957
958    /// Cases excluded or flagged under the honest edge.
959    pub excluded: Vec<UnevaluableCase>,
960
961    /// The validity-boundary statement. Always [`TWIN_REPORT_BOUND`];
962    /// carried as a field so the constraint travels with the evidence.
963    pub bound: String,
964}
965
966/// The result of one twin run: the journaled evidence, the decision stream,
967/// the metrics, and the report.
968#[derive(Debug)]
969pub struct TwinOutcome {
970    /// The twin run's journal snapshot (super-step boundaries, served and
971    /// faulted effect events, and every `PolicyDecision`).
972    pub journal: JournalSnapshot,
973
974    /// Every decision the run journaled — acting and shadow interleaved by
975    /// decision point, in decision order.
976    pub decisions: Vec<DecisionEvent>,
977
978    /// Aggregate metrics.
979    pub metrics: TwinMetrics,
980
981    /// The validity report.
982    pub report: TwinReport,
983}
984
985/// A directed what-if: at `decision_seq`, apply `action` instead of the
986/// acting policy's choice, then continue.
987#[derive(Debug)]
988pub struct CounterfactualFork {
989    /// The twin run's decision sequence number to fork at (a report's
990    /// `shadow_divergences` are the natural candidates).
991    pub decision_seq: u64,
992
993    /// The different legal action to apply. Must be a member of the legal
994    /// set recomputed at that decision point — the twin refuses anything
995    /// else, which is where the honest edge bites: no legal set contains an
996    /// input-changing action.
997    pub action: DecisionAction,
998
999    /// Hybrid replay (R0.5's deferred mode): after the fork, act with this
1000    /// policy instead of the run's original acting policy. `None` keeps the
1001    /// original acting policy — a single-decision what-if.
1002    pub then_act_with: Option<Arc<dyn TwinPolicy>>,
1003}
1004
1005/// The result of a counterfactual branch: the branch's own outcome plus its
1006/// [`BranchDiff`] against the baseline — fork-plus-diff, comparison as
1007/// journaled evidence.
1008#[derive(Debug)]
1009pub struct CounterfactualBranch {
1010    /// The branch run's outcome.
1011    pub outcome: TwinOutcome,
1012
1013    /// The baseline's outcome the branch was diffed against.
1014    pub baseline: TwinOutcome,
1015
1016    /// The logical diff: the fork's divergence point, the work each side
1017    /// did after it, step-level differences, and per-side totals.
1018    pub diff: BranchDiff,
1019}
1020
1021// ---------------------------------------------------------------------------
1022// The twin.
1023// ---------------------------------------------------------------------------
1024
1025/// A deterministic re-execution environment over one recorded run.
1026///
1027/// Construction verifies the journal's integrity through the replay
1028/// module's boundary ([`ExactReplay::new`]) and decomposes the servable
1029/// effects into the world the scheduler re-drives. The typical flow:
1030///
1031/// 1. `let twin = Twin::from_fixture(&fixture)?;`
1032/// 2. `let baseline = twin.run(&TwinRunConfig::new(seed))?;` — the
1033///    recorded world under the floor;
1034/// 3. probe it: `twin.run(&config.with_faults(schedule))`,
1035///    `twin.run_interleavings(8, &config.with_seeded_interleaving())`,
1036///    `twin.run(&config.with_shadow(candidate))`, or
1037///    `twin.counterfactual(&config, fork)`.
1038#[derive(Debug)]
1039pub struct Twin {
1040    world: TwinWorld,
1041    run_id: String,
1042    thread_id: String,
1043}
1044
1045impl Twin {
1046    /// A twin over an integrity-verified snapshot. Resumed-run journals are
1047    /// rejected with the same boundary exact replay applies: their evidence
1048    /// begins mid-run against checkpointed state the journal does not
1049    /// carry.
1050    pub fn from_snapshot(snapshot: JournalSnapshot) -> Result<Self> {
1051        // Integrity verification (head hash, artifacts, references, the
1052        // resumed-run boundary) is the replay module's contract, consumed —
1053        // not re-implemented.
1054        ExactReplay::new(snapshot.clone())?;
1055        let world = TwinWorld::from_snapshot(&snapshot)?;
1056        Ok(Self {
1057            world,
1058            run_id: snapshot.run_id,
1059            thread_id: snapshot.thread_id,
1060        })
1061    }
1062
1063    /// A twin over a portable fixture's journal: a recorded production run
1064    /// becomes a twin case by export, unmodified.
1065    pub fn from_fixture(fixture: &ReplayFixture) -> Result<Self> {
1066        Self::from_snapshot(fixture.journal.clone())
1067    }
1068
1069    /// The recorded world this twin re-executes against.
1070    pub fn world(&self) -> &TwinWorld {
1071        &self.world
1072    }
1073
1074    /// The twin run id for `seed` — deterministic, so equal configs share
1075    /// an id and their journals compare byte-for-byte.
1076    fn twin_run_id(&self, seed: u64) -> String {
1077        format!("twin:{}:{seed:016x}", self.run_id)
1078    }
1079
1080    /// Re-execute the recorded world under `config`.
1081    pub fn run(&self, config: &TwinRunConfig) -> Result<TwinOutcome> {
1082        self.run_inner(config, None)
1083    }
1084
1085    /// Re-run the recorded world under `runs` seeded interleavings
1086    /// (mechanism 2). Run `k` draws its permutation and jitter from
1087    /// `config.seed` mixed with `k`, so the interleaving distribution is
1088    /// sampled deterministically and each point of it reproduces exactly.
1089    /// The journaled total order of evidence is canonical in every run —
1090    /// what varies is admission order and therefore per-node latencies.
1091    pub fn run_interleavings(
1092        &self,
1093        runs: usize,
1094        config: &TwinRunConfig,
1095    ) -> Result<Vec<TwinOutcome>> {
1096        (0..runs)
1097            .map(|k| {
1098                let run_config = TwinRunConfig {
1099                    seed: config.seed ^ (k as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15),
1100                    clock: config.clock,
1101                    faults: config.faults.clone(),
1102                    acting: config.acting.clone(),
1103                    shadow: config.shadow.clone(),
1104                    timeout_ladder: config.timeout_ladder.clone(),
1105                    concurrency_ladder: config.concurrency_ladder.clone(),
1106                    workers: config.workers.clone(),
1107                    max_attempts: config.max_attempts,
1108                    interleaving: Interleaving::Seeded,
1109                };
1110                self.run(&run_config)
1111            })
1112            .collect()
1113    }
1114
1115    /// Fork the recorded run at one decision: `baseline` behavior up to the
1116    /// fork, `fork.action` applied at it, then `fork.then_act_with` (or the
1117    /// original acting policy) afterward — with the same fault schedule
1118    /// still in force.
1119    ///
1120    /// Refuses, with [`UnevaluableCase`] naming the case, when the fork's
1121    /// action is outside the legal set recomputed at its decision point
1122    /// (the honest edge made structural) or when the run never reaches the
1123    /// named decision.
1124    pub fn counterfactual(
1125        &self,
1126        baseline: &TwinRunConfig,
1127        fork: CounterfactualFork,
1128    ) -> Result<CounterfactualBranch> {
1129        let base_outcome = self.run(baseline)?;
1130        let branch_outcome = self.run_inner(baseline, Some(&fork))?;
1131        let diff = BranchDiff::between(&base_outcome.journal, &branch_outcome.journal);
1132        Ok(CounterfactualBranch {
1133            outcome: branch_outcome,
1134            baseline: base_outcome,
1135            diff,
1136        })
1137    }
1138
1139    /// The shared run machinery. With `fork = Some`, the decision at
1140    /// `fork.decision_seq` is overridden (validated against its recomputed
1141    /// legal set) and the acting policy swaps afterward.
1142    fn run_inner(
1143        &self,
1144        config: &TwinRunConfig,
1145        fork: Option<&CounterfactualFork>,
1146    ) -> Result<TwinOutcome> {
1147        if config.workers.is_empty() {
1148            return Err(twin_error(
1149                "the worker pool is empty; placement needs at least one equivalent worker",
1150            ));
1151        }
1152        if config
1153            .timeout_ladder
1154            .iter()
1155            .any(|rung| *rung < MIN_TIMEOUT_RUNG_MS)
1156        {
1157            return Err(twin_error(format!(
1158                "timeout ladder carries a rung below the {MIN_TIMEOUT_RUNG_MS} ms minimum — \
1159                 below it ordinary work aborts early, a correctness hazard no policy may cross"
1160            )));
1161        }
1162        let run_id = self.twin_run_id(config.seed);
1163        let mut state = RunState {
1164            journal: Journal::new(
1165                run_id.clone(),
1166                self.thread_id.clone(),
1167                Clock::logical(config.clock.start_ms, config.clock.tick_ms),
1168            ),
1169            rng: RngSource::seeded(config.seed),
1170            run_id,
1171            decisions: Vec::new(),
1172            pending: Vec::new(),
1173            next_decision_seq: 0,
1174            faults_fired: 0,
1175            divergences: Vec::new(),
1176            excluded: Vec::new(),
1177            forked: false,
1178        };
1179        // The acting policy is swappable at exactly one point: the fork.
1180        let mut acting = config.acting.clone();
1181        // Simulated worker lanes: free-at times in milliseconds since run
1182        // start. Steps are barriers; a step starts when every lane is free.
1183        let mut lane_free = vec![0u64; config.workers.len()];
1184        let mut item_elapsed: Vec<u64> = Vec::new();
1185        let mut metrics = TwinMetrics {
1186            items: self.world.items.len(),
1187            ..TwinMetrics::default()
1188        };
1189
1190        for &step in self.world.steps() {
1191            let items = self.world.items_in(step);
1192            if items.is_empty() {
1193                continue;
1194            }
1195            let step_start_time = lane_free.iter().max().copied().unwrap_or(0);
1196            let step_event = state.journal.record(
1197                EventDraft::new(RunEventKind::SuperStepStart, Effect::Pure).input(
1198                    serde_json::json!({
1199                        "step": step,
1200                        "active_nodes": items
1201                            .iter()
1202                            .map(|item| item.node_id.clone().unwrap_or_else(|| format!("seq:{}", item.recorded_seq)))
1203                            .collect::<Vec<_>>(),
1204                    }),
1205                ),
1206            );
1207
1208            // The concurrency decision for this step: how many admissions
1209            // the policy allows at once, bounded above by the pool (a
1210            // learned limit may only narrow what the pool admits).
1211            let mut features = Map::new();
1212            features.insert("step".to_owned(), Value::from(step));
1213            features.insert("queue_depth".to_owned(), Value::from(items.len()));
1214            features.insert("workers".to_owned(), Value::from(config.workers.len()));
1215            let legal: Vec<DecisionAction> = config
1216                .concurrency_ladder
1217                .iter()
1218                .map(|limit| DecisionAction::SetConcurrency { limit: *limit })
1219                .collect();
1220            let action = decide(
1221                &mut state,
1222                fork,
1223                &mut acting,
1224                config,
1225                DecisionFamily::Concurrency,
1226                features,
1227                legal,
1228                &step_event,
1229            )?;
1230            let admitted = match action {
1231                DecisionAction::SetConcurrency { limit } => {
1232                    (limit as usize).min(config.workers.len()).max(1)
1233                }
1234                _ => 1,
1235            };
1236            flush_pending(&mut state);
1237
1238            // Mechanism 2: the execution order of the parallel task set.
1239            // What the order changes is which item waits behind which on
1240            // the admitted lanes; the journaled order stays canonical
1241            // because each item's drafts are buffered and flushed in
1242            // recorded order after the step executes.
1243            let mut order: Vec<usize> = (0..items.len()).collect();
1244            if config.interleaving == Interleaving::Seeded {
1245                for i in (1..order.len()).rev() {
1246                    let j = (state.rng.next_f64() * (i + 1) as f64) as usize;
1247                    order.swap(i, j.min(i));
1248                }
1249            }
1250
1251            let mut step_outcomes: BTreeMap<String, &str> = BTreeMap::new();
1252            let mut buffers: Vec<Vec<EventDraft>> = vec![Vec::new(); items.len()];
1253            for &position in &order {
1254                let item = items[position];
1255                let key = format!(
1256                    "{}#{}",
1257                    item.node_id.as_deref().unwrap_or("seq"),
1258                    item.recorded_seq
1259                );
1260                let outcome = run_item(
1261                    &mut state,
1262                    fork,
1263                    &mut acting,
1264                    config,
1265                    item,
1266                    admitted,
1267                    &mut lane_free,
1268                    &step_event,
1269                    step_start_time,
1270                )?;
1271                buffers[position] = std::mem::take(&mut state.pending);
1272                metrics.attempts += outcome.attempts;
1273                metrics.cost_usd += outcome.cost_usd;
1274                item_elapsed.push(outcome.elapsed_ms);
1275                let disposition = match outcome.terminal() {
1276                    Terminal::Completed => {
1277                        metrics.completed += 1;
1278                        "completed"
1279                    }
1280                    Terminal::DeadLettered => {
1281                        metrics.dead_lettered += 1;
1282                        "dead_lettered"
1283                    }
1284                    Terminal::Failed => {
1285                        metrics.failed += 1;
1286                        "failed"
1287                    }
1288                };
1289                step_outcomes.insert(key, disposition);
1290            }
1291
1292            // Canonical flush: the journal's total order of evidence is
1293            // stable across interleavings — execution order only moved the
1294            // timing attributes (admission waits, latencies, timestamps).
1295            for buffer in &mut buffers {
1296                for draft in buffer.drain(..) {
1297                    state.journal.record(draft);
1298                }
1299            }
1300
1301            // The step's end evidence is the per-item dispositions — the
1302            // twin's analogue of the recorded journal's channel values, and
1303            // what BranchDiff's step diffs read.
1304            state.journal.record(
1305                EventDraft::new(RunEventKind::SuperStepEnd, Effect::Pure)
1306                    .output(serde_json::to_value(&step_outcomes)?)
1307                    .parent(step_event),
1308            );
1309        }
1310
1311        metrics.wall_time_ms = lane_free.iter().max().copied().unwrap_or(0);
1312        metrics.item_latency_p50_ms = percentile(&item_elapsed, 50.0);
1313        metrics.item_latency_p95_ms = percentile(&item_elapsed, 95.0);
1314
1315        if let Some(fork) = fork {
1316            if !state.forked {
1317                let case = UnevaluableCase::UnknownDecision {
1318                    decision_seq: fork.decision_seq,
1319                };
1320                return Err(twin_error(format!(
1321                    "counterfactual refused: the run never reaches decision d{} — {}",
1322                    fork.decision_seq,
1323                    serde_json::to_string(&case).unwrap_or_else(|_| format!("{case:?}"))
1324                )));
1325            }
1326        }
1327
1328        let journal = state.journal.snapshot();
1329        let report = TwinReport {
1330            run_id: state.run_id.clone(),
1331            seed: config.seed,
1332            faults_declared: config.faults.injections.len(),
1333            faults_fired: state.faults_fired,
1334            decisions: state.next_decision_seq as usize,
1335            evaluable_decisions: state.next_decision_seq as usize,
1336            shadow_divergences: state.divergences,
1337            excluded: state.excluded,
1338            bound: TWIN_REPORT_BOUND.to_owned(),
1339        };
1340        Ok(TwinOutcome {
1341            journal,
1342            decisions: state.decisions,
1343            metrics,
1344            report,
1345        })
1346    }
1347}
1348
1349/// How a work item ended.
1350#[derive(Debug, Clone, Copy, PartialEq, Eq)]
1351enum Terminal {
1352    Completed,
1353    DeadLettered,
1354    Failed,
1355}
1356
1357/// What one work item cost the run.
1358#[derive(Debug, Default)]
1359struct ItemOutcome {
1360    terminal: Option<Terminal>,
1361    attempts: u64,
1362    elapsed_ms: u64,
1363    cost_usd: f64,
1364}
1365
1366impl ItemOutcome {
1367    fn terminal(&self) -> Terminal {
1368        // Every path out of `run_item` sets the terminal state; defaulting
1369        // to `Failed` here would hide a machinery bug as evidence.
1370        self.terminal.expect("run_item always settles its item")
1371    }
1372}
1373
1374/// Mutable per-run evidence state: the journal being written, the decision
1375/// stream, the current item's buffered drafts, and the counters the report
1376/// is built from.
1377struct RunState {
1378    journal: Journal,
1379    rng: RngSource,
1380    run_id: String,
1381    decisions: Vec<DecisionEvent>,
1382    /// Drafts of the item currently executing, flushed in canonical order
1383    /// at the step boundary (see `run_inner`); keeps the journaled total
1384    /// order stable under seeded interleavings.
1385    pending: Vec<EventDraft>,
1386    next_decision_seq: u64,
1387    faults_fired: usize,
1388    divergences: Vec<u64>,
1389    excluded: Vec<UnevaluableCase>,
1390    /// `true` once the fork decision has been taken (a fork that never
1391    /// fires is the `UnknownDecision` refusal).
1392    forked: bool,
1393}
1394
1395/// Write every buffered draft to the journal in order.
1396fn flush_pending(state: &mut RunState) {
1397    for draft in std::mem::take(&mut state.pending) {
1398        state.journal.record(draft);
1399    }
1400}
1401
1402/// One decision point: assemble the evidence, let the acting policy (and
1403/// the shadow) score the same features, validate both against the contract,
1404/// journal both, and return the action that executes.
1405///
1406/// Validation is the policy plane's enforcement at the twin boundary: a
1407/// selection outside `legal_actions` or a propensity outside `(0, 1]` is a
1408/// contract violation, rejected rather than journaled — dishonest evidence
1409/// is worse than no evidence.
1410#[allow(clippy::too_many_arguments)]
1411fn decide(
1412    state: &mut RunState,
1413    fork: Option<&CounterfactualFork>,
1414    acting: &mut Arc<dyn TwinPolicy>,
1415    config: &TwinRunConfig,
1416    family: DecisionFamily,
1417    features: Map<String, Value>,
1418    legal_actions: Vec<DecisionAction>,
1419    parent: &str,
1420) -> Result<DecisionAction> {
1421    let seq = state.next_decision_seq;
1422    let ctx = DecisionContext {
1423        family,
1424        features: &features,
1425        legal_actions: &legal_actions,
1426    };
1427
1428    // Mechanism 3: the fork overrides exactly one decision. The override
1429    // must be legal *here* — the legal set is recomputed from the same
1430    // gates, so an input-changing or gate-crossing action is refused
1431    // against evidence, not against the experimenter's say-so.
1432    let at_fork = fork.is_some_and(|fork| fork.decision_seq == seq);
1433    let (selected, propensity, version) = if at_fork {
1434        let fork = fork.expect("checked above");
1435        if !legal_actions.contains(&fork.action) {
1436            let case = UnevaluableCase::IllegalForkAction {
1437                decision_seq: seq,
1438                action: fork.action.clone(),
1439            };
1440            return Err(twin_error(format!(
1441                "counterfactual refused: {:?} is not in the legal set at decision d{seq} \
1442                 ({legal_actions:?}) — {}",
1443                fork.action,
1444                serde_json::to_string(&case).unwrap_or_else(|_| format!("{case:?}"))
1445            )));
1446        }
1447        state.forked = true;
1448        (
1449            fork.action.clone(),
1450            1.0,
1451            PolicyVersion::new(TWIN_FORK_POLICY_VERSION),
1452        )
1453    } else {
1454        let (selected, propensity) = acting.decide(&ctx, state.rng.next_f64());
1455        validate_decision(&selected, propensity, &legal_actions, &acting.version())?;
1456        (selected, propensity, acting.version())
1457    };
1458
1459    let decided_at = state.journal.clock().now();
1460    let acting_event = DecisionEvent {
1461        id: format!("{}:d{seq}", state.run_id),
1462        run_id: state.run_id.clone(),
1463        thread_id: state.journal.thread_id().to_owned(),
1464        seq,
1465        family,
1466        features: features.clone(),
1467        legal_actions: legal_actions.clone(),
1468        selected: selected.clone(),
1469        propensity,
1470        policy_version: version,
1471        role: Some(DecisionRole::Acting),
1472        outcome: None,
1473        decided_at,
1474    };
1475    journal_decision(state, &acting_event, parent);
1476
1477    // Mechanism 4: the shadow scores the same features and journals its
1478    // decision with its true propensity; it never acts. Divergence from
1479    // the acting decision is recorded — those are the points a
1480    // counterfactual branch estimates the candidate's outcome from.
1481    if let Some(shadow) = &config.shadow {
1482        let (shadow_selected, shadow_propensity) = shadow.decide(&ctx, state.rng.next_f64());
1483        validate_decision(
1484            &shadow_selected,
1485            shadow_propensity,
1486            &legal_actions,
1487            &shadow.version(),
1488        )?;
1489        let shadow_event = DecisionEvent {
1490            id: format!("{}:d{seq}:shadow", state.run_id),
1491            selected: shadow_selected.clone(),
1492            propensity: shadow_propensity,
1493            policy_version: shadow.version(),
1494            role: Some(DecisionRole::Shadow),
1495            decided_at: state.journal.clock().now(),
1496            ..acting_event.clone()
1497        };
1498        journal_decision(state, &shadow_event, parent);
1499        if shadow_selected != selected {
1500            state.divergences.push(seq);
1501        }
1502    }
1503
1504    state.next_decision_seq += 1;
1505
1506    // Hybrid replay: after the fork decision, the new policy acts.
1507    if at_fork {
1508        if let Some(next) = fork.and_then(|fork| fork.then_act_with.as_ref()) {
1509            *acting = next.clone();
1510        }
1511    }
1512    Ok(selected)
1513}
1514
1515/// The policy-plane contract, enforced where decisions are journaled.
1516fn validate_decision(
1517    selected: &DecisionAction,
1518    propensity: f64,
1519    legal_actions: &[DecisionAction],
1520    version: &PolicyVersion,
1521) -> Result<()> {
1522    if !legal_actions.contains(selected) {
1523        return Err(twin_error(format!(
1524            "policy `{}` selected {selected:?}, which is outside the legal set \
1525             {legal_actions:?} — a policy chooses among declared members, never free-form \
1526             outputs",
1527            version.as_str()
1528        )));
1529    }
1530    if !(0.0 < propensity && propensity <= 1.0) {
1531        return Err(twin_error(format!(
1532            "policy `{}` declared propensity {propensity}, outside (0, 1] — propensity is \
1533             assigned at decision time and must be truthful, or off-policy evaluation over \
1534             this evidence is meaningless",
1535            version.as_str()
1536        )));
1537    }
1538    Ok(())
1539}
1540
1541/// Buffer one decision as a `PolicyDecision` draft (flushed in canonical
1542/// order at the step boundary) and keep it in the run's decision stream.
1543fn journal_decision(state: &mut RunState, event: &DecisionEvent, parent: &str) {
1544    state.pending.push(
1545        EventDraft::new(RunEventKind::PolicyDecision, Effect::Pure)
1546            .output(
1547                serde_json::to_value(event)
1548                    .unwrap_or_else(|_| serde_json::json!({"id": event.id, "error": "decision payload failed to serialize"})),
1549            )
1550            .parent(parent),
1551    );
1552    state.decisions.push(event.clone());
1553}
1554
1555/// What one attempt was observed to do — after faults, timeout bounds, and
1556/// the recorded world's answer are composed. What the policy observes is
1557/// not always what the world did: a bound truncates the observation, the
1558/// truth stays.
1559struct AttemptObservation {
1560    status: EventStatus,
1561    class: Option<ErrorClass>,
1562    retry_after_ms: Option<u64>,
1563    /// Observed attempt latency, excluding admission wait.
1564    latency_ms: u64,
1565    output: Option<Value>,
1566    tokens: Option<Usage>,
1567    cost_usd: Option<f64>,
1568}
1569
1570/// Drive one work item to its terminal state: attempts, faults, decision
1571/// points, and the retry loop, against the simulated worker lanes.
1572#[allow(clippy::too_many_arguments)]
1573fn run_item(
1574    state: &mut RunState,
1575    fork: Option<&CounterfactualFork>,
1576    acting: &mut Arc<dyn TwinPolicy>,
1577    config: &TwinRunConfig,
1578    item: &TwinWorkItem,
1579    admitted: usize,
1580    lane_free: &mut [u64],
1581    step_event: &str,
1582    step_start_time: u64,
1583) -> Result<ItemOutcome> {
1584    let mut outcome = ItemOutcome::default();
1585    let item_ready = step_start_time;
1586    let mut prev_attempt_end = item_ready;
1587    let mut attempt = 0u32;
1588
1589    loop {
1590        attempt += 1;
1591        outcome.attempts += 1;
1592
1593        // Placement: which equivalent worker this attempt runs on. Decided
1594        // per attempt — a retry may re-place, which is exactly what the
1595        // family would learn.
1596        let mut features = Map::new();
1597        features.insert("attempt".to_owned(), Value::from(attempt));
1598        features.insert("effect".to_owned(), serde_json::to_value(item.effect)?);
1599        features.insert(
1600            "lane_free_ms".to_owned(),
1601            serde_json::to_value(&*lane_free)?,
1602        );
1603        if let Some(node) = &item.node_id {
1604            features.insert("node".to_owned(), Value::from(node.clone()));
1605        }
1606        let legal: Vec<DecisionAction> = config
1607            .workers
1608            .iter()
1609            .map(|worker| DecisionAction::SelectWorker {
1610                worker: worker.clone(),
1611            })
1612            .collect();
1613        let action = decide(
1614            state,
1615            fork,
1616            acting,
1617            config,
1618            DecisionFamily::WorkerPlacement,
1619            features,
1620            legal,
1621            step_event,
1622        )?;
1623        let worker = match &action {
1624            DecisionAction::SelectWorker { worker } => worker.clone(),
1625            _ => config.workers[0].clone(),
1626        };
1627        let lane = config
1628            .workers
1629            .iter()
1630            .position(|w| w == &worker)
1631            .unwrap_or(0)
1632            // The concurrency cap admits only so many lanes at once; a
1633            // placement onto a lane beyond the cap queues behind it.
1634            .min(admitted.saturating_sub(1))
1635            .min(lane_free.len() - 1);
1636
1637        // Timeout: the bound this attempt runs under.
1638        let mut features = Map::new();
1639        features.insert(
1640            "recorded_latency_ms".to_owned(),
1641            Value::from(item.answer.latency_ms),
1642        );
1643        features.insert("attempt".to_owned(), Value::from(attempt));
1644        features.insert("effect".to_owned(), serde_json::to_value(item.effect)?);
1645        if let Some(node) = &item.node_id {
1646            features.insert("node".to_owned(), Value::from(node.clone()));
1647        }
1648        let legal: Vec<DecisionAction> = config
1649            .timeout_ladder
1650            .iter()
1651            .map(|millis| DecisionAction::SetTimeout { millis: *millis })
1652            .collect();
1653        let action = decide(
1654            state,
1655            fork,
1656            acting,
1657            config,
1658            DecisionFamily::Timeout,
1659            features,
1660            legal,
1661            step_event,
1662        )?;
1663        let bound = match action {
1664            DecisionAction::SetTimeout { millis } => millis,
1665            _ => MAX_RETRY_DELAY_MS,
1666        };
1667
1668        // The effect boundary: the fault schedule is consulted the way the
1669        // production scheduler reads the world — first matching anchor
1670        // wins, and a fired fault replaces the recorded answer.
1671        let fault = config
1672            .faults
1673            .fault_for_attempt(item.recorded_seq, attempt, &worker)
1674            .cloned();
1675        let mut observation = match &fault {
1676            Some(fault) => {
1677                state.faults_fired += 1;
1678                faulted_observation(fault, bound, item)
1679            }
1680            None => recorded_observation(item, bound),
1681        };
1682
1683        // A crash at the decision point: the scheduler's fail-task path
1684        // observes the in-flight attempt through the injected fault (lease
1685        // expiry classifies `Unknown`). Consulted only where the production
1686        // scheduler would see a lost attempt — a retry decision about to be
1687        // made.
1688        if observation.status == EventStatus::Error {
1689            if let Some(at_decision) = config
1690                .faults
1691                .fault_for_decision(state.next_decision_seq)
1692                .cloned()
1693            {
1694                state.faults_fired += 1;
1695                observation = faulted_observation(&at_decision, bound, item);
1696            }
1697        }
1698
1699        // Admission on the lane: the attempt starts when the lane frees and
1700        // any retry delay has elapsed; the journaled latency is what the
1701        // run experiences — the wait plus the observed attempt.
1702        let ready = if attempt == 1 {
1703            item_ready
1704        } else {
1705            prev_attempt_end
1706        };
1707        let start = ready.max(lane_free[lane]);
1708        let wait = start.saturating_sub(ready);
1709        let end = start + observation.latency_ms;
1710        lane_free[lane] = end;
1711        let journaled_latency = wait + observation.latency_ms;
1712
1713        // Buffer the attempt as the recorded kind with the observed
1714        // outcome (flushed in canonical order at the step boundary).
1715        // Retried attempts re-issue the same bytes — the twin never
1716        // rewrites an effect's input, which is the honest edge made
1717        // structural.
1718        let mut draft = EventDraft::new(item.kind, item.effect)
1719            .input(item.request.clone())
1720            .latency_ms(journaled_latency)
1721            .status(observation.status)
1722            .parent(step_event);
1723        if let Some(node) = &item.node_id {
1724            draft = draft.node(node.clone());
1725        }
1726        match observation.status {
1727            EventStatus::Ok => {
1728                if let Some(output) = &observation.output {
1729                    draft = draft.output(output.clone());
1730                }
1731                if let Some(tokens) = observation.tokens {
1732                    draft = draft.tokens(tokens);
1733                }
1734            }
1735            EventStatus::Error => {
1736                let class = observation.class.unwrap_or(ErrorClass::Unknown);
1737                draft = draft.output(serde_json::json!({
1738                    "error": serde_json::to_value(class)?,
1739                }));
1740            }
1741            EventStatus::Interrupted => unreachable!("servable calls never interrupt"),
1742        }
1743        if let Some(cost) = observation.cost_usd {
1744            draft = draft.cost_usd(cost);
1745            outcome.cost_usd += cost;
1746        }
1747        state.pending.push(draft);
1748        prev_attempt_end = end;
1749
1750        if observation.status == EventStatus::Ok {
1751            outcome.terminal = Some(Terminal::Completed);
1752            outcome.elapsed_ms = end.saturating_sub(item_ready);
1753            return Ok(outcome);
1754        }
1755
1756        // The retry decision: the scheduler's fail-task path. The legal set
1757        // comes from the runtime's own gates — effect gate, class gate,
1758        // attempt budget — so no policy, learned or otherwise, can route
1759        // around them.
1760        let class = observation.class.unwrap_or(ErrorClass::Unknown);
1761        if fault.is_some() && !item.effect.is_freely_repeatable() {
1762            // The fault's future is the gate's, not the policy's: flag it
1763            // so the report names what bounded the evaluation.
1764            let flag = UnevaluableCase::GatedEffect {
1765                effect_seq: item.recorded_seq,
1766            };
1767            if !state.excluded.contains(&flag) {
1768                state.excluded.push(flag);
1769            }
1770        }
1771        let mut features = Map::new();
1772        features.insert("failure_class".to_owned(), serde_json::to_value(class)?);
1773        features.insert("attempt".to_owned(), Value::from(attempt));
1774        features.insert("max_attempts".to_owned(), Value::from(config.max_attempts));
1775        features.insert("effect".to_owned(), serde_json::to_value(item.effect)?);
1776        features.insert(
1777            "dependency_latency_ms".to_owned(),
1778            Value::from(journaled_latency),
1779        );
1780        if let Some(retry_after) = observation.retry_after_ms {
1781            features.insert("retry_after_ms".to_owned(), Value::from(retry_after));
1782        }
1783        let legal = retry_legal_actions(item.effect, class, attempt, config.max_attempts);
1784        // Dead-letter is the retryable-budget-exhausted case: the legal set
1785        // collapses to Abort while the gates would still have allowed a
1786        // retry had budget remained.
1787        let budget_exhausted = item.effect.is_freely_repeatable()
1788            && class.is_retryable()
1789            && attempt >= config.max_attempts;
1790        let action = decide(
1791            state,
1792            fork,
1793            acting,
1794            config,
1795            DecisionFamily::Retry,
1796            features,
1797            legal,
1798            step_event,
1799        )?;
1800        match action {
1801            DecisionAction::Retry { .. } => {
1802                // The delay is policy-parameterized within declared bounds,
1803                // floored by the callee's Retry-After — the one delay the
1804                // world imposes.
1805                let delay = acting
1806                    .retry_delay_ms_for(Some(class), attempt, state.rng.next_f64())
1807                    .max(observation.retry_after_ms.unwrap_or(0));
1808                lane_free[lane] = lane_free[lane].max(prev_attempt_end + delay);
1809                prev_attempt_end += delay;
1810            }
1811            _ => {
1812                outcome.terminal = Some(if budget_exhausted {
1813                    Terminal::DeadLettered
1814                } else {
1815                    Terminal::Failed
1816                });
1817                outcome.elapsed_ms = end.saturating_sub(item_ready);
1818                return Ok(outcome);
1819            }
1820        }
1821    }
1822}
1823
1824/// The recorded world's answer, observed through the timeout bound: a
1825/// completing attempt slower than its bound is observed as `Timeout` at the
1826/// bound — the policy's observation changes, the world's truth does not.
1827fn recorded_observation(item: &TwinWorkItem, bound: u64) -> AttemptObservation {
1828    match item.answer.status {
1829        EventStatus::Ok if item.answer.latency_ms > bound => AttemptObservation {
1830            status: EventStatus::Error,
1831            class: Some(ErrorClass::Timeout),
1832            retry_after_ms: None,
1833            latency_ms: bound,
1834            output: None,
1835            tokens: None,
1836            cost_usd: item.answer.cost_usd,
1837        },
1838        _ => AttemptObservation {
1839            status: item.answer.status,
1840            class: (item.answer.status == EventStatus::Error).then_some(ErrorClass::Unknown),
1841            retry_after_ms: None,
1842            latency_ms: item.answer.latency_ms,
1843            output: item.answer.output.clone(),
1844            tokens: item.answer.tokens,
1845            cost_usd: item.answer.cost_usd,
1846        },
1847    }
1848}
1849
1850/// The injected fault as an observation. Latencies follow the world's
1851/// rules, not convenience: a crash is discovered at the lease boundary, a
1852/// hang surfaces at the tighter of the bound and the boundary, and
1853/// fail-fast faults answer at the recorded latency (the attempt ran, then
1854/// the world said no).
1855fn faulted_observation(
1856    fault: &InjectedFault,
1857    bound: u64,
1858    item: &TwinWorkItem,
1859) -> AttemptObservation {
1860    let (latency_ms, retry_after_ms) = match fault {
1861        InjectedFault::WorkerCrash => (MAX_RETRY_DELAY_MS, None),
1862        InjectedFault::CalleeTimeout => (bound.min(MAX_RETRY_DELAY_MS), None),
1863        InjectedFault::RateLimited { retry_after_ms } => {
1864            (item.answer.latency_ms, Some(*retry_after_ms))
1865        }
1866        InjectedFault::ResourceExhausted => (item.answer.latency_ms, None),
1867    };
1868    AttemptObservation {
1869        status: EventStatus::Error,
1870        class: Some(fault.error_class()),
1871        retry_after_ms,
1872        latency_ms,
1873        output: None,
1874        tokens: None,
1875        // Unpriced by the recording, so unpriced here: the attempt count
1876        // is the honest cost proxy, the same accounting Wave 1's
1877        // durable-work class uses.
1878        cost_usd: None,
1879    }
1880}
1881
1882/// Nearest-rank percentile of an unsorted sample set. `pub(crate)`: the
1883/// drift detector (`crate::learn`) reads the same statistic off journaled
1884/// decision features — one nearest-rank definition, shared.
1885pub(crate) fn percentile(samples: &[u64], p: f64) -> u64 {
1886    if samples.is_empty() {
1887        return 0;
1888    }
1889    let mut sorted = samples.to_vec();
1890    sorted.sort_unstable();
1891    let rank = ((p / 100.0) * sorted.len() as f64).ceil() as usize;
1892    sorted[rank.max(1).min(sorted.len()) - 1]
1893}
1894
1895#[cfg(test)]
1896mod tests {
1897    use super::*;
1898    use serde_json::json;
1899
1900    fn ctx<'a>(
1901        family: DecisionFamily,
1902        features: &'a Map<String, Value>,
1903        legal: &'a [DecisionAction],
1904    ) -> DecisionContext<'a> {
1905        DecisionContext {
1906            family,
1907            features,
1908            legal_actions: legal,
1909        }
1910    }
1911
1912    #[test]
1913    fn fault_schedule_matches_in_declaration_order() {
1914        let schedule = FaultSchedule::new(7)
1915            .with_injection(
1916                FaultAnchor::Window {
1917                    from_seq: 0,
1918                    to_seq: 10,
1919                },
1920                InjectedFault::ResourceExhausted,
1921            )
1922            .with_injection(
1923                FaultAnchor::OnAttempt {
1924                    effect_seq: 3,
1925                    attempt: 1,
1926                },
1927                InjectedFault::RateLimited {
1928                    retry_after_ms: 500,
1929                },
1930            )
1931            .with_injection(
1932                FaultAnchor::OnWorker {
1933                    worker: "w1".to_owned(),
1934                },
1935                InjectedFault::WorkerCrash,
1936            );
1937
1938        // The window is declared first and wins for seq 3 despite the more
1939        // specific OnAttempt entry — declaration order is the precedence
1940        // rule, and it is deterministic.
1941        assert_eq!(
1942            schedule.fault_for_attempt(3, 1, "w0"),
1943            Some(&InjectedFault::ResourceExhausted)
1944        );
1945        // Outside the window: the specific anchors answer.
1946        assert_eq!(
1947            schedule.fault_for_attempt(11, 1, "w1"),
1948            Some(&InjectedFault::WorkerCrash)
1949        );
1950        assert_eq!(schedule.fault_for_attempt(11, 1, "w0"), None);
1951        // AtDecision anchors answer only the decision query.
1952        assert_eq!(schedule.fault_for_decision(0), None);
1953        let schedule = schedule.with_injection(
1954            FaultAnchor::AtDecision { decision_seq: 2 },
1955            InjectedFault::WorkerCrash,
1956        );
1957        assert_eq!(
1958            schedule.fault_for_decision(2),
1959            Some(&InjectedFault::WorkerCrash)
1960        );
1961        // Outside the window and off the degraded worker: nothing fires.
1962        assert_eq!(schedule.fault_for_attempt(11, 2, "w0"), None);
1963    }
1964
1965    #[test]
1966    fn injected_faults_classify_into_the_retry_taxonomy() {
1967        assert_eq!(
1968            InjectedFault::WorkerCrash.error_class(),
1969            ErrorClass::Unknown
1970        );
1971        assert_eq!(
1972            InjectedFault::CalleeTimeout.error_class(),
1973            ErrorClass::Timeout
1974        );
1975        assert_eq!(
1976            InjectedFault::RateLimited { retry_after_ms: 50 }.error_class(),
1977            ErrorClass::RateLimited
1978        );
1979        assert_eq!(
1980            InjectedFault::ResourceExhausted.error_class(),
1981            ErrorClass::ResourceExhausted
1982        );
1983    }
1984
1985    #[test]
1986    fn floor_decides_the_static_v0_stance_per_family() {
1987        let features = Map::new();
1988        let floor = StaticFloor;
1989
1990        // Retry: retries while the gates leave a Retry member legal.
1991        let legal = vec![DecisionAction::Retry { attempt: 2 }, DecisionAction::Abort];
1992        let (action, propensity) =
1993            floor.decide(&ctx(DecisionFamily::Retry, &features, &legal), 0.5);
1994        assert_eq!(action, DecisionAction::Retry { attempt: 2 });
1995        assert_eq!(propensity, 1.0);
1996
1997        // Gates closed: only Abort remains.
1998        let legal = vec![DecisionAction::Abort];
1999        let (action, _) = floor.decide(&ctx(DecisionFamily::Retry, &features, &legal), 0.5);
2000        assert_eq!(action, DecisionAction::Abort);
2001
2002        // Timeout: the top rung — "no bound in force" modeled as the lease
2003        // boundary, matching Wave 1's accounting.
2004        let legal: Vec<DecisionAction> = DEFAULT_TIMEOUT_LADDER
2005            .iter()
2006            .map(|millis| DecisionAction::SetTimeout { millis: *millis })
2007            .collect();
2008        let (action, _) = floor.decide(&ctx(DecisionFamily::Timeout, &features, &legal), 0.5);
2009        assert_eq!(
2010            action,
2011            DecisionAction::SetTimeout {
2012                millis: MAX_RETRY_DELAY_MS
2013            }
2014        );
2015
2016        // Concurrency: uncapped. Placement: the first eligible worker (the
2017        // static-pool stance).
2018        let legal = vec![
2019            DecisionAction::SetConcurrency { limit: 1 },
2020            DecisionAction::SetConcurrency { limit: u32::MAX },
2021        ];
2022        let (action, _) = floor.decide(&ctx(DecisionFamily::Concurrency, &features, &legal), 0.5);
2023        assert_eq!(action, DecisionAction::SetConcurrency { limit: u32::MAX });
2024        let legal = vec![
2025            DecisionAction::SelectWorker {
2026                worker: "w0".to_owned(),
2027            },
2028            DecisionAction::SelectWorker {
2029                worker: "w1".to_owned(),
2030            },
2031        ];
2032        let (action, _) = floor.decide(
2033            &ctx(DecisionFamily::WorkerPlacement, &features, &legal),
2034            0.5,
2035        );
2036        assert_eq!(
2037            action,
2038            DecisionAction::SelectWorker {
2039                worker: "w0".to_owned()
2040            }
2041        );
2042    }
2043
2044    #[test]
2045    fn validate_decision_rejects_illegal_actions_and_bad_propensities() {
2046        let legal = vec![DecisionAction::Abort];
2047        let version = PolicyVersion::new("policy-test");
2048        assert!(validate_decision(&DecisionAction::Abort, 1.0, &legal, &version).is_ok());
2049        let error = validate_decision(&DecisionAction::Retry { attempt: 2 }, 1.0, &legal, &version)
2050            .unwrap_err();
2051        assert!(error.to_string().contains("outside the legal set"));
2052        for bad in [0.0, -0.5, 1.5] {
2053            assert!(validate_decision(&DecisionAction::Abort, bad, &legal, &version).is_err());
2054        }
2055    }
2056
2057    #[test]
2058    fn timeout_bound_truncates_the_observation_not_the_world() {
2059        let item = TwinWorkItem {
2060            recorded_seq: 0,
2061            kind: RunEventKind::ToolCall,
2062            node_id: Some("n".to_owned()),
2063            effect: Effect::Idempotent,
2064            request: json!({"tool": "t", "arguments": {}}),
2065            answer: RecordedAnswer {
2066                status: EventStatus::Ok,
2067                output: Some(json!({"r": 1})),
2068                latency_ms: 10_000,
2069                tokens: None,
2070                cost_usd: Some(0.01),
2071            },
2072            step: 0,
2073        };
2074        // Slower than the bound: observed as Timeout at the bound.
2075        let observed = recorded_observation(&item, 5_000);
2076        assert_eq!(observed.status, EventStatus::Error);
2077        assert_eq!(observed.class, Some(ErrorClass::Timeout));
2078        assert_eq!(observed.latency_ms, 5_000);
2079        // Within the bound: the recorded answer, untouched.
2080        let observed = recorded_observation(&item, 30_000);
2081        assert_eq!(observed.status, EventStatus::Ok);
2082        assert_eq!(observed.latency_ms, 10_000);
2083        assert_eq!(observed.output, Some(json!({"r": 1})));
2084    }
2085
2086    #[test]
2087    fn faulted_observations_follow_the_worlds_latency_rules() {
2088        let item = TwinWorkItem {
2089            recorded_seq: 0,
2090            kind: RunEventKind::ToolCall,
2091            node_id: None,
2092            effect: Effect::Idempotent,
2093            request: json!({}),
2094            answer: RecordedAnswer {
2095                status: EventStatus::Ok,
2096                output: None,
2097                latency_ms: 2_000,
2098                tokens: None,
2099                cost_usd: None,
2100            },
2101            step: 0,
2102        };
2103        // A crash is discovered at the lease boundary.
2104        let crash = faulted_observation(&InjectedFault::WorkerCrash, 5_000, &item);
2105        assert_eq!(crash.latency_ms, MAX_RETRY_DELAY_MS);
2106        // A hang surfaces at the tighter of bound and boundary.
2107        let hang = faulted_observation(&InjectedFault::CalleeTimeout, 5_000, &item);
2108        assert_eq!(hang.latency_ms, 5_000);
2109        let hang = faulted_observation(&InjectedFault::CalleeTimeout, MAX_RETRY_DELAY_MS, &item);
2110        assert_eq!(hang.latency_ms, MAX_RETRY_DELAY_MS);
2111        // Rate limits carry their Retry-After floor.
2112        let limited = faulted_observation(
2113            &InjectedFault::RateLimited {
2114                retry_after_ms: 800,
2115            },
2116            5_000,
2117            &item,
2118        );
2119        assert_eq!(limited.retry_after_ms, Some(800));
2120        assert_eq!(limited.latency_ms, 2_000);
2121    }
2122
2123    #[test]
2124    fn percentile_is_nearest_rank() {
2125        assert_eq!(percentile(&[], 50.0), 0);
2126        assert_eq!(percentile(&[10], 95.0), 10);
2127        assert_eq!(percentile(&[1, 2, 3, 4], 50.0), 2);
2128        assert_eq!(percentile(&[1, 2, 3, 4], 95.0), 4);
2129    }
2130}