moonpool-sim 0.8.0

Simulation engine for the moonpool framework
Documentation
//! # Moonpool Simulation Framework
//!
//! Deterministic simulation for testing distributed systems, inspired by
//! [FoundationDB's simulation testing](https://apple.github.io/foundationdb/testing.html).
//!
//! ## Why Deterministic Simulation?
//!
//! `FoundationDB`'s insight: **bugs hide in error paths**. Production code rarely
//! exercises timeout handlers, retry logic, or failure recovery. Deterministic
//! simulation with fault injection finds these bugs before production does.
//!
//! Key properties:
//! - **Reproducible**: Same seed produces identical execution
//! - **Comprehensive**: Tests all failure modes (network, timing, corruption)
//! - **Fast**: Logical time skips idle periods
//!
//! ## Core Components
//!
//! - [`SimWorld`]: The simulation runtime managing events and time
//! - [`SimulationBuilder`]: Configure and run simulations
//! - [`chaos`]: Fault injection (buggify, 14 assertion macros, invariants)
//! - [`storage`]: Storage simulation with fault injection
//! - Multiverse exploration via `moonpool-explorer` (re-exported as [`ExplorationConfig`], [`AdaptiveConfig`])
//!
//! ## Fault Injection Overview
//!
//! See [`chaos`] module for detailed documentation.
//!
//! | Mechanism | Default | What it tests |
//! |-----------|---------|---------------|
//! | TCP latencies | 1-11ms connect | Async scheduling |
//! | Random connection close | 0.001% | Reconnection, redelivery |
//! | Bit flip corruption | 0.01% | Checksum validation |
//! | Connect failure | 50% probabilistic | Timeout handling, retries |
//! | Clock drift | 100ms max | Leases, heartbeats |
//! | Buggified delays | 25% | Race conditions |
//! | Partial writes | 1000 bytes max | Message fragmentation |
//! | Packet loss | disabled | At-least-once delivery |
//! | Network partitions | disabled | Split-brain handling |
//! | Storage corruption | configurable | Checksum validation, recovery |
//! | Torn writes | configurable | Write atomicity, journaling |
//! | Sync failures | configurable | Durability guarantees |
//!
//! ## Coverage-Preserving Multi-Seed Exploration
//!
//! When exploration is enabled, multiple seeds share coverage context. The
//! explored map (coverage bitmap union) and assertion watermarks are preserved
//! between seeds so each subsequent seed focuses energy on genuinely new
//! branches rather than re-treading already-discovered paths.
//!
//! A **warm start** mechanism reduces wasted forks: on seeds after the first,
//! marks whose first probe batch finds zero new coverage bits stop after
//! `warm_min_timelines` forks instead of the full `min_timelines`.
//!
//! ```text
//!  Seed 1 (cold start)           Seed 2 (warm start)          Seed 3 (warm start)
//!  energy: 400K                  energy: 400K                 energy: 400K
//!
//!  root ─┬─ mark A ──> 400       root ─┬─ mark A ──> 30      root ─┬─ mark A ──> 30
//!        │  (new bits!) forks           │  (barren!) skip           │  (barren!) skip
//!        │                              │                           │
//!        ├─ mark B ──> 400              ├─ mark B ──> 30            ├─ mark B ──> 30
//!        │  (new bits!) forks           │  (barren!) skip           │  (barren!) skip
//!        │                              │                           │
//!        └─ mark C ──> 400              ├─ mark C ──> 30            ├─ mark C ──> 30
//!           (new bits!) forks           │  (barren!) skip           │  (barren!) skip
//!                                       │                           │
//!                                       └─ mark D ──> 400          └─ mark E ──> 400
//!                                          (NEW bits!) forks           (NEW bits!) forks
//!                        │                               │                            │
//!       ┌────────────────┘              ┌────────────────┘           ┌────────────────┘
//!       v                               v                            v
//!  ┌──────────┐  ──preserved──>  ┌──────────┐  ──preserved──>  ┌──────────┐
//!  │ explored │  coverage map    │ explored │  coverage map    │ explored │
//!  │ map:     │  + watermarks    │ map:     │  + watermarks    │ map:     │
//!  │ A,B,C    │                  │ A,B,C,D  │                  │ A,B,C,D,E│
//!  └──────────┘                  └──────────┘                  └──────────┘
//!
//!  Total: each seed spends most energy on NEW discoveries.
//!  Warm marks (A,B,C on seed 2) exit after warm_min_timelines (30)
//!  instead of min_timelines (400), saving ~95% energy per barren mark.
//! ```
//!
//! ```ignore
//! SimulationBuilder::new()
//!     .set_iterations(3)  // 3 root seeds with coverage forwarding
//!     .enable_exploration(ExplorationConfig {
//!         max_depth: 120,
//!         timelines_per_split: 4,
//!         global_energy: 400_000,  // per-seed energy budget
//!         adaptive: Some(AdaptiveConfig {
//!             batch_size: 30,
//!             min_timelines: 400,
//!             max_timelines: 1_000,
//!             per_mark_energy: 10_000,
//!             warm_min_timelines: Some(30),  // quick skip for barren warm marks
//!         }),
//!         parallelism: Some(Parallelism::HalfCores),
//!     })
//!     .workload(my_workload);
//! ```

#![deny(missing_docs)]
#![deny(clippy::unwrap_used)]

// Re-export core types for convenience
pub use moonpool_core::{
    CodecError, Endpoint, JsonCodec, MessageCodec, NetworkAddress, NetworkAddressParseError,
    NetworkProvider, Providers, RandomProvider, SimulationError, SimulationResult, TaskProvider,
    TcpListenerTrait, TimeError, TimeProvider, UID, WELL_KNOWN_RESERVED_COUNT, WellKnownToken,
};
// The deterministic select! (moonpool-sim always enables core's
// deterministic-select, so this is tokio's expansion with a seeded start
// offset). Process and
// workload code must use this instead of tokio::select!, whose branch offset is
// entropy-drawn outside a seeded tokio runtime.
pub use moonpool_core::select;
// Production provider bundle — only when the tokio-providers feature pulls core's
// net/fs/time. A wasm-able sim (`--no-default-features`) omits these.
#[cfg(feature = "tokio-providers")]
pub use moonpool_core::{
    TokioNetworkProvider, TokioProviders, TokioTaskProvider, TokioTimeProvider,
};

// =============================================================================
// Core Modules
// =============================================================================

/// Core simulation engine for deterministic testing.
pub mod sim;

/// The deterministic single-threaded executor (seeded-random scheduling).
pub mod executor;

/// Simulation runner and orchestration framework.
pub mod runner;

/// Chaos testing infrastructure for deterministic fault injection.
pub mod chaos;

/// Provider implementations for simulation.
pub mod providers;

/// Network simulation and configuration.
pub mod network;

/// Production-friendly observability layer (replaces legacy Timeline + Invariant).
pub mod observability;

/// Storage simulation and configuration.
pub mod storage;

/// Simulation workloads and binary targets.
pub mod simulations;

// =============================================================================
// Public API Re-exports
// =============================================================================

// Sim module re-exports
pub use sim::{
    ConnectionStateChange, Event, EventQueue, NetworkOperation, ScheduledEvent, SimFaultRecord,
    SimWorld, SleepFuture, StorageOperation, WeakSimWorld, clear_rng_breakpoints, current_sim_seed,
    reset_rng_call_count, reset_sim_rng, rng_call_count, set_rng_breakpoints, set_sim_seed,
    set_swarm_op_seed, sim_random, sim_random_range, sim_random_range_or_default, swarm_op_enabled,
};

// Runner module re-exports
pub use runner::{
    Attrition, AttritionScope, Chaos, ChaosMode, ClientId, DomainLevel, FaultContext,
    FaultInjector, IterationControl, LocalityConfig, LocalityInfo, MachineRegistry, Process,
    ProcessTags, RebootKind, SimContext, SimulationBuilder, SimulationMetrics, SimulationReport,
    TagRegistry, Workload, WorkloadCount, WorkloadTopology,
};

// Chaos module re-exports
pub use chaos::{
    AssertionStats, SIM_FAULT_EVENT_NAME, SimFaultEvent, StateHandle, assertion_results,
    buggify_init, buggify_reset, has_always_violations, reset_always_violations,
    reset_assertion_results, validate_assertion_contracts,
};

// Observability module re-exports (plain-tracing capture + invariants)
pub use observability::{
    Clock, FieldValue, InstallGuard, Invariant, SimTime, SimulationLayer, SimulationLayerHandle,
    TraceEvent, TraceQuery, init_sim_tracing, invariant_fn,
};

// Network exports
pub use network::{
    ChaosConfiguration, ConnectFailureMode, LatencyDistribution, NetworkConfiguration,
    SimNetworkProvider, sample_duration, sample_latency,
};

// Storage exports
pub use storage::{
    InMemoryStorage, SECTOR_SIZE, SectorBitSet, SimStorageProvider, StorageConfiguration,
    StorageError,
};

// Provider exports
pub use providers::{SimProviders, SimRandomProvider, SimTaskProvider, SimTimeProvider};

// Assertion vocabulary — always available (dependency-free accounting layer).
pub use moonpool_assertions::{AssertCmp, AssertKind};
// Exploration-only re-exports (fork-based multiverse engine).
#[cfg(feature = "exploration")]
pub use moonpool_explorer::{
    AdaptiveConfig, ExplorationConfig, Parallelism, format_timeline, parse_timeline,
};
pub use runner::report::{BugRecipe, ExplorationReport};

// Macros are automatically available at crate root when defined with #[macro_export]