Struct SimulationBuilder 

pub struct SimulationBuilder { /* private fields */ }

Builder pattern for configuring and running simulation experiments.

Implementations

impl SimulationBuilder

pub fn new() -> Self

Create a new empty simulation builder.

pub fn workload(self, w: impl Workload) -> Self

Add a single workload instance to the simulation.

The instance is reused across iterations (the run() method is called each iteration on the same struct). Gets client_id = 0, client_count = 1.

pub fn processes( self, count: impl Into<ProcessCount>, factory: impl Fn() -> Box<dyn Process> + 'static, ) -> Self

Add server processes to the simulation.

Processes represent the system under test — they can be killed and restarted (rebooted). A fresh instance is created from the factory on every boot.

The count parameter accepts either a fixed usize or a RangeInclusive<usize> for seeded random count per iteration.

Only one .processes() call is supported per builder. Subsequent calls overwrite the previous one.

Examples

// Fixed 3 server processes
builder.processes(3, || Box::new(MyNode::new()))

// 3 to 7 processes, randomized per iteration
builder.processes(3..=7, || Box::new(MyNode::new()))

pub fn cluster( self, config: LocalityConfig, factory: impl Fn() -> Box<dyn Process> + 'static, ) -> Self

Register server processes laid out across a failure-domain topology.

Unlike processes, the LocalityConfig is the spawn spec: it determines the process count (sampled per seed), assigns each process a datacenter / zone / machine, and lets machine- and zone-scoped attrition reboot collocated processes together. Calling this replaces any prior .processes() / .cluster() registration.

Tags (tags) remain orthogonal and may still be chained.

Examples

// 3 datacenters × 3 zones × 3 machines × 1 process = 27 processes,
// with the datacenter count randomized per seed.
builder.cluster(
    LocalityConfig::new(1..=3, 3, 3, 1),
    || Box::new(MyNode::new()),
)

pub fn tags( self, dimensions: &[(&str, &[&str])], ) -> Result<Self, SimulationError>

Attach tag distribution to the last .processes() call.

Tags are distributed round-robin across process instances. Each tag dimension is distributed independently.

Errors

Returns SimulationError::InvalidState if called without a preceding .processes() call.

Examples

// 5 processes: dc cycles east/west/eu, rack cycles r1/r2
builder.processes(5, || Box::new(MyNode::new()))
    .tags(&[
        ("dc", &["east", "west", "eu"]),
        ("rack", &["r1", "r2"]),
    ])?

pub fn attrition(self, config: Attrition) -> Self

Set built-in attrition for automatic process reboots during chaos phase.

Attrition randomly kills and restarts server processes. It respects max_dead to limit the number of simultaneously dead processes.

Requires .chaos_duration() — attrition injectors only run during the chaos phase. Without a chaos duration, the injector will not be spawned.

For custom fault injection, use .fault() with a FaultInjector instead.

pub fn workloads( self, count: WorkloadCount, factory: impl Fn(usize) -> Box<dyn Workload> + 'static, ) -> Self

Add multiple workload instances from a factory.

The factory receives an instance index (0-based) and must return a fresh workload. Instances are created each iteration and dropped afterward. Client IDs default to sequential starting from 0 (FDB-style).

The workload is responsible for its own name() — use the index to produce unique names when count > 1 (e.g., format!("client-{i}")).

Examples

// 3 fixed replicas
builder.workloads(WorkloadCount::Fixed(3), |i| Box::new(ReplicaWorkload::new(i)))

// 1–5 random clients
builder.workloads(WorkloadCount::Random(1..6), |i| Box::new(ClientWorkload::new(i)))

pub fn invariant<I: Invariant>(self, i: I) -> Self

Add an invariant to be checked after every simulation step.

pub fn invariant_fn( self, name: impl Into<String>, f: impl Fn(&dyn TraceQuery, u64) + Send + 'static, ) -> Self

Add a closure-based invariant.

pub fn fault(self, f: impl FaultInjector) -> Self

Add a fault injector to run during the chaos phase.

pub fn chaos_duration(self, duration: Duration) -> Self

Set the chaos phase duration.

When set, fault injectors run concurrently with workloads for this duration. After it elapses, faults stop and the system continues until all workloads complete. A settle phase then drains remaining events before checks run.

pub fn set_iterations(self, iterations: usize) -> Self

Set the number of iterations to run.

pub fn seed_warning_timeout(self, timeout: Duration) -> Self

Set the wall-clock time threshold for warning about slow seeds.

When a seed takes longer than this duration, a tracing::warn! is emitted. If not set, no slow-seed warnings are produced.

pub fn run_time_budget(self, budget: Duration) -> Self

Set the virtual-time budget for a single run phase.

If simulated time advances past this bound while one or more workloads are still running, the orchestrator first triggers a graceful shutdown and — if simulated time keeps climbing by another full budget while workloads remain — declares the run deadlocked.

This is a deterministic safety net for a self-perpetuating timer: a detached task (e.g. a reconnect / keepalive loop) that re-arms a crate::TimeProvider::sleep every tick keeps the event queue non-empty forever, so the no-progress deadlock detector never fires even though no workload-relevant progress is being made. The budget turns that silent hang into an actionable deadlock failure.

The decision is a pure function of the simulated event schedule (no wall clock, no RNG), so it never perturbs replay determinism. The default (one simulated hour) is deliberately generous; raise it for legitimately long simulations.

pub fn until_coverage_stable( self, plateau_seeds: usize, max_iterations: usize, ) -> Self

Run until the system is saturated: every observed assert_sometimes! / assert_reachable! assertion has fired and code coverage has not grown for plateau_seeds consecutive seeds (capped at max_iterations).

Uses real LLVM sancov code coverage when the binary is instrumented (built via cargo xtask sim run); otherwise falls back to assertion-slot coverage. Works with or without SimulationBuilder::enable_exploration; no fork occurs unless exploration is explicitly enabled. e.g. until_coverage_stable(10, 5000).

pub fn before_iteration(self, f: impl FnMut() + 'static) -> Self

Register a callback invoked at the start of each simulation iteration.

Use this to reset shared state (directories, membership, stores) that lives outside the builder and is shared via Rc across iterations.

pub fn set_debug_seeds(self, seeds: Vec<u64>) -> Self

Set specific seeds for deterministic debugging and regression testing.

pub fn enable_chaos(self, surfaces: impl IntoIterator<Item = Chaos>) -> Self

Enable chaos surfaces and choose how each is sampled per seed.

Each Chaos entry turns on one surface (network / storage / attrition) in a given ChaosMode — Random (full surface every seed) or Swarm (a per-seed random subset of sub-families, the rest fully off). A surface not listed stays off. Later entries override earlier ones for the same surface.

Swarm mode defeats passive suppression: when every fault is always slightly on (Random) families crowd each other out and the extreme single-family configs that surface bugs almost never occur. Subset decisions are drawn from a dedicated CONFIG_RNG stream, so they are reproducible per seed yet never perturb in-run randomness or fork-explorer replay.

The workload operation-alphabet swarm is a separate, test-driver concern — see swarm_operations.

Example

builder.enable_chaos([
    Chaos::Network(ChaosMode::Swarm),
    Chaos::Storage(ChaosMode::Swarm),
]);

pub fn swarm_operations(self) -> Self

Enable per-seed swarm testing of the workload operation alphabet.

When enabled, each seed exposes a random subset of each workload’s operation alphabet via swarm_op_enabled, so bugs reachable only when whole operation groups are suppressed become reachable across seeds. Decisions come from a dedicated per-seed stream and are reproducible. Independent of enable_chaos.

pub fn enable_exploration(self, config: ExplorationConfig) -> Self

Enable fork-based multiverse exploration.

When enabled, the simulation will fork child processes at assertion discovery points to explore alternate timelines with different seeds. Requires the exploration feature.

pub fn run(self) -> SimulationReport

Run the simulation and generate a report.

Creates a fresh deterministic Executor per iteration for full isolation — all tasks are killed when the executor is dropped at iteration end.

Panics

Panics if a simulation invariant fails or a workload panics.

Trait Implementations

impl Default for SimulationBuilder

fn default() -> Self

Returns the “default value” for a type.