Crate moonpool_sim 

Moonpool Simulation Framework

Deterministic simulation for testing distributed systems, inspired by FoundationDB’s simulation testing.

Why Deterministic Simulation?

FoundationDB’s insight: bugs hide in error paths. Production code rarely exercises timeout handlers, retry logic, or failure recovery. Deterministic simulation with fault injection finds these bugs before production does.

Key properties:

• Reproducible: Same seed produces identical execution
• Comprehensive: Tests all failure modes (network, timing, corruption)
• Fast: Logical time skips idle periods

Core Components

• SimWorld: The simulation runtime managing events and time
• SimulationBuilder: Configure and run simulations
• chaos: Fault injection (buggify, assertions, invariants)

Quick Start

use moonpool_sim::{SimulationBuilder, WorkloadTopology};

SimulationBuilder::new()
    .topology(WorkloadTopology::ClientServer { clients: 2, servers: 1 })
    .run(|ctx| async move {
        // Your distributed system workload
    });

Fault Injection Overview

See chaos module for detailed documentation.

Mechanism	Default	What it tests
TCP latencies	1-11ms connect	Async scheduling
Random connection close	0.001%	Reconnection, redelivery
Bit flip corruption	0.01%	Checksum validation
Connect failure	50% probabilistic	Timeout handling, retries
Clock drift	100ms max	Leases, heartbeats
Buggified delays	25%	Race conditions
Partial writes	1000 bytes max	Message fragmentation
Packet loss	disabled	At-least-once delivery
Network partitions	disabled	Split-brain handling

Multi-Seed Testing

Tests run across multiple seeds to explore the state space:

SimulationBuilder::new()
    .run_count(IterationControl::UntilAllSometimesReached(1000))
    .run(workload);

Debugging a failing seed:

SimulationBuilder::new()
    .set_seed(failing_seed)
    .run_count(IterationControl::FixedCount(1))
    .run(workload);

Re-exports

pub use sim::ConnectionStateChange;

pub use sim::Event;

pub use sim::EventQueue;

pub use sim::NetworkOperation;

pub use sim::ScheduledEvent;

pub use sim::SimWorld;

pub use sim::SleepFuture;

pub use sim::WeakSimWorld;

pub use sim::get_current_sim_seed;

pub use sim::reset_sim_rng;

pub use sim::set_sim_seed;

pub use sim::sim_random;

pub use sim::sim_random_range;

pub use sim::sim_random_range_or_default;

pub use runner::IterationControl;

pub use runner::SimulationBuilder;

pub use runner::SimulationMetrics;

pub use runner::SimulationReport;

pub use runner::TokioReport;

pub use runner::TokioRunner;

pub use runner::WorkloadTopology;

pub use chaos::AssertionStats;

pub use chaos::InvariantCheck;

pub use chaos::StateRegistry;

pub use chaos::buggify_init;

pub use chaos::buggify_reset;

pub use chaos::get_assertion_results;

pub use chaos::panic_on_assertion_violations;

pub use chaos::reset_assertion_results;

pub use chaos::validate_assertion_contracts;

pub use network::ChaosConfiguration;

pub use network::ConnectFailureMode;

pub use network::NetworkConfiguration;

pub use network::SimNetworkProvider;

pub use network::sample_duration;

pub use providers::SimRandomProvider;

pub use providers::SimTimeProvider;

Modules

chaos

Chaos testing infrastructure for deterministic fault injection. Chaos testing infrastructure for deterministic fault injection.

network

Network simulation and configuration. Network simulation and configuration.

providers

Provider implementations for simulation. Provider implementations for simulation.

runner

Simulation runner and orchestration framework. Simulation runner and orchestration framework.

sim

Core simulation engine for deterministic testing. Core simulation engine for deterministic testing.

Macros

always_assert

Assert that a condition is always true, panicking on failure.

buggify

Buggify with 25% probability

buggify_with_prob

Buggify with custom probability

sometimes_assert

Assert a condition that should sometimes be true, tracking the success rate.

Structs

Endpoint

Endpoint = Address + Token (FDB-compatible).

JsonCodec

JSON codec using serde_json.

NetworkAddress

Network address (IPv4/IPv6 + port + flags).

TokioNetworkProvider

Real Tokio networking implementation.

TokioTaskProvider

Tokio-based task provider using spawn_local for single-threaded execution.

TokioTimeProvider

Real time provider using Tokio’s time facilities.

UID

128-bit unique identifier (FDB-compatible).

Enums

CodecError

Error type for codec operations.

NetworkAddressParseError

Error parsing a network address from string.

SimulationError

Errors that can occur during simulation operations.

WellKnownToken

Well-known endpoint tokens (FDB-compatible pattern).

Constants

WELL_KNOWN_RESERVED_COUNT

Number of reserved well-known token slots.

Traits

MessageCodec

Pluggable message serialization format.

NetworkProvider

Provider trait for creating network connections and listeners.

RandomProvider

Provider trait for random number generation.

TaskProvider

Provider for spawning local tasks in single-threaded context.

TcpListenerTrait

Trait for TCP listeners that can accept connections.

TimeProvider

Provider trait for time operations.

Type Aliases

SimulationResult

A type alias for Result<T, SimulationError>.