Expand description
§Moonpool
Deterministic simulation testing for distributed systems in Rust.
Moonpool enables you to write distributed system logic once, test it with simulated networking for reproducible debugging, then deploy with real networking—all using identical application code.
Inspired by FoundationDB’s simulation testing.
§Crate Architecture
┌─────────────────────────────────────────────────────────────┐
│ moonpool (this crate) │
│ Re-exports all functionality + virtual actors module │
├──────────────────────────┬──────────────────────────────────┤
│ moonpool-transport │ moonpool-sim │
│ • Peer connections │ • SimWorld runtime │
│ • Wire format │ • Chaos testing │
│ • NetTransport + RPC │ • Buggify macros │
│ • #[service] macro │ • 14 assertion macros │
│ (via transport-derive)│ • Multiverse exploration │
│ │ (via moonpool-explorer) │
├──────────────────────────┴──────────────────────────────────┤
│ moonpool-core │
│ Provider traits: Time, Task, Network, Random, Storage │
│ Core types: UID, Endpoint, NetworkAddress │
└─────────────────────────────────────────────────────────────┘§Quick Start
ⓘ
use moonpool::{SimulationBuilder, WorkloadTopology};
SimulationBuilder::new()
.topology(WorkloadTopology::ClientServer { clients: 2, servers: 1 })
.run(|ctx| async move {
// Your distributed system workload
});§Which Crate to Use
| Use case | Crate |
|---|---|
| Full framework (recommended) | moonpool |
| Provider traits only | moonpool-core |
| Simulation without transport | moonpool-sim |
| Transport without simulation | moonpool-transport |
| Fork-based exploration internals | moonpool-explorer |
| Proc-macro internals | moonpool-transport-derive |
§Documentation
moonpool_core- Provider traits and core typesmoonpool_sim- Simulation runtime and chaos testingmoonpool_transport- Network transport layeractors- Virtual actor system (Orleans-style)
Modules§
- actors
- Virtual actor networking layer for moonpool.
- chaos
- Chaos testing infrastructure for deterministic fault injection. Chaos testing infrastructure for deterministic fault injection.
- error
- Error types for transport operations. Error types for the moonpool messaging layer.
- flags
- Address flags.
- network
- Network simulation and configuration. Network simulation and configuration.
- peer
- Resilient peer connection management. Resilient peer connection management.
- providers
- Provider implementations for simulation. Provider implementations for simulation.
- rpc
- RPC layer with typed request/response patterns. FDB-style static messaging with fixed endpoints.
- runner
- Simulation runner and orchestration framework. Simulation runner and orchestration framework.
- sim
- Core simulation engine for deterministic testing. Core simulation engine for deterministic testing.
- simulations
- Simulation workloads for chaos testing and exploration.
- storage
- Storage simulation and configuration. Storage simulation and configuration.
- wire
- Wire format with CRC32C checksums. Wire format for packet serialization.
Macros§
- assert_
always - Assert that a condition is always true.
- assert_
always_ greater_ than - Assert that
val > thresholdalways holds. - assert_
always_ greater_ than_ or_ equal_ to - Assert that
val >= thresholdalways holds. - assert_
always_ less_ than - Assert that
val < thresholdalways holds. - assert_
always_ less_ than_ or_ equal_ to - Assert that
val <= thresholdalways holds. - assert_
always_ or_ unreachable - Assert that a condition is always true when reached, but the code path need not be reached. Does not panic if never evaluated.
- assert_
reachable - Assert that a code path is reachable (should be reached at least once).
- assert_
sometimes - Assert a condition that should sometimes be true, tracking stats and triggering exploration.
- assert_
sometimes_ all - Compound boolean assertion: all named bools should sometimes be true simultaneously.
- assert_
sometimes_ each - Per-value bucketed sometimes assertion with optional quality watermarks.
- assert_
sometimes_ greater_ than - Assert that
val > thresholdsometimes holds. Forks on watermark improvement. - assert_
sometimes_ greater_ than_ or_ equal_ to - Assert that
val >= thresholdsometimes holds. Forks on watermark improvement. - assert_
sometimes_ less_ than - Assert that
val < thresholdsometimes holds. Forks on watermark improvement. - assert_
sometimes_ less_ than_ or_ equal_ to - Assert that
val <= thresholdsometimes holds. Forks on watermark improvement. - assert_
unreachable - Assert that a code path should never be reached.
- buggify
- Buggify with 25% probability
- buggify_
with_ prob - Buggify with custom probability
Structs§
- Adaptive
Config - Configuration for adaptive batch-based timeline splitting.
- Assertion
Stats - Statistics for a tracked assertion.
- Attrition
- Built-in attrition configuration for automatic process reboots.
- BugRecipe
- A captured bug recipe with its root seed for deterministic replay.
- Chaos
Configuration - Configuration for chaos injection in simulations.
- Endpoint
- Endpoint = Address + Token.
- Endpoint
Map - Maps endpoint tokens to message receivers.
- Event
Queue - A priority queue for scheduling events in chronological order.
- Exploration
Config - Configuration for exploration.
- Exploration
Report - Report from fork-based exploration.
- Failure
Monitor - Reactive failure monitor for address and endpoint tracking.
- Fault
Context - Context for fault injectors — gives access to SimWorld fault injection methods.
- InMemory
Storage - In-memory storage with deterministic fault injection.
- Json
Codec - JSON codec using serde_json.
- Monitor
Config - Configuration for connection health monitoring.
- NetNotified
Queue - Type-safe message queue with async notification.
- NetTransport
- Central transport coordinator (FDB NetTransport equivalent).
- NetTransport
Builder - Builder for NetTransport that eliminates common footguns.
- Network
Address - Network address (IPv4/IPv6 + port + flags).
- Network
Configuration - Configuration for network simulation parameters
- Node
Address - Identifies a specific node instance in the cluster.
- Open
Options - Options for opening a file.
- Packet
Header - Packet header for wire format.
- Peer
- A resilient peer that manages connections to a remote address.
- Peer
Config - Configuration for peer behavior and reconnection parameters.
- Peer
Metrics - Metrics and state information for a peer connection.
- Phase
Config - Two-phase simulation configuration.
- Process
Tags - Resolved tags for a specific process instance.
- Reply
Future - Future that resolves when a reply is received from the server.
- Reply
Promise - Promise for sending a reply to a request.
- Request
Envelope - Envelope wrapping a request with its reply endpoint.
- Request
Stream - Stream for receiving typed requests with reply promises.
- Scheduled
Event - An event scheduled for execution at a specific simulation time.
- Sector
BitSet - A simple bitset for tracking sector states.
- Server
Handle - Handle for a running RPC server.
- SimContext
- Simulation context provided to workloads.
- SimNetwork
Provider - Simulated networking implementation
- SimProviders
- Simulation providers bundle for deterministic testing.
- SimRandom
Provider - Random provider for simulation that uses the thread-local deterministic RNG.
- SimStorage
Provider - Simulated storage provider for deterministic testing.
- SimTime
Provider - Simulation time provider that integrates with SimWorld.
- SimWorld
- The central simulation coordinator that manages time and event processing.
- Simulation
Builder - Builder pattern for configuring and running simulation experiments.
- Simulation
Metrics - Core metrics collected during a simulation run.
- Simulation
Report - Comprehensive report of a simulation run with statistical analysis.
- Sleep
Future - Future that completes after a specified simulation time duration.
- State
Handle - Shared state handle for cross-workload communication and invariant checking.
- Storage
Configuration - Configuration for storage simulation parameters.
- TagRegistry
- Registry mapping process IPs to their resolved tags.
- Tokio
Network Provider - Real Tokio networking implementation.
- Tokio
Providers - Production providers using Tokio runtime.
- Tokio
Random Provider - Production random provider using thread-local RNG.
- Tokio
Report - Report generated after running workloads with TokioRunner.
- Tokio
Runner - Builder for executing workloads with real Tokio implementations.
- Tokio
Storage File - Wrapper for Tokio File to implement our trait.
- Tokio
Storage Provider - Real Tokio storage implementation.
- Tokio
Task Provider - Tokio-based task provider using spawn_local for single-threaded execution.
- Tokio
TcpListener - Wrapper for Tokio TcpListener to implement our trait.
- Tokio
Time Provider - Real time provider using Tokio’s time facilities.
- UID
- 128-bit unique identifier.
- Weak
SimWorld - A weak reference to a simulation world.
- Workload
Topology - Topology information provided to workloads and processes.
Enums§
- Assert
Cmp - Comparison operator for numeric assertions.
- Assert
Kind - The kind of assertion being tracked.
- Client
Id - Strategy for assigning client IDs to workload instances.
- Codec
Error - Error type for codec operations.
- Connect
Failure Mode - Connection establishment failure mode for fault injection.
- Connection
State Change - Connection state changes
- Event
- Events that can be scheduled in the simulation.
- Failed
Reason - Reason an endpoint was permanently marked as failed.
- Failure
Status - Status of a network address or endpoint.
- Iteration
Control - Configuration for how many iterations a simulation should run.
- Messaging
Error - Errors that can occur in the messaging layer.
- Network
Address Parse Error - Error parsing a network address from string.
- Network
Operation - Network data operations
- Parallelism
- Controls how many children can run in parallel during splitting.
- Peer
Error - Errors that can occur during peer operations.
- Reboot
Kind - The type of reboot to perform on a process.
- Reply
Error - Errors that can occur during request-response operations.
- RpcError
- Unified error type for RPC operations.
- Simulation
Error - Errors that can occur during simulation operations.
- Storage
Operation - Storage operations that can be scheduled in the simulation.
- Time
Error - Errors that can occur during time operations.
- Well
Known Token - Well-known endpoint tokens.
- Wire
Error - Wire format error types.
- Workload
Count - How many instances of a workload to spawn per iteration.
Constants§
- HEADER_
SIZE - Header size: 4 (length) + 4 (checksum) + 16 (token) = 24 bytes.
- MAX_
PAYLOAD_ SIZE - Maximum payload size (1MB).
- SECTOR_
SIZE - Size of a disk sector in bytes.
- WELL_
KNOWN_ RESERVED_ COUNT - Number of reserved well-known token slots.
Traits§
- Fault
Injector - A fault injector that introduces failures during the chaos phase.
- Invariant
- An invariant that validates system-wide properties during simulation.
- Message
Codec - Pluggable message serialization format.
- Message
Receiver - Trait for receiving deserialized messages from the transport layer.
- Network
Provider - Provider trait for creating network connections and listeners.
- Process
- A process that participates in simulation as part of the system under test.
- Providers
- Bundle of all provider types for a runtime environment.
- Random
Provider - Provider trait for random number generation.
- Storage
File - Trait for file handles that support async read/write/seek operations.
- Storage
Provider - Provider trait for file storage operations.
- Task
Provider - Provider for spawning local tasks in single-threaded context.
- TcpListener
Trait - Trait for TCP listeners that can accept connections.
- Time
Provider - Provider trait for time operations.
- Workload
- A workload that participates in simulation testing.
Functions§
- buggify_
init - Initialize buggify for simulation run
- buggify_
reset - Reset/disable buggify
- clear_
rng_ breakpoints - Clear all RNG breakpoints.
- deserialize_
packet - Deserialize a packet, validating checksum.
- format_
timeline - Format a recipe as a human-readable timeline string.
- get_
assertion_ results - Get current assertion statistics for all tracked assertions.
- get_
current_ sim_ seed - Get the current simulation seed.
- get_
reply - At-least-once delivery: send reliably, retransmit on reconnect.
- get_
reply_ unless_ failed_ for - At-least-once delivery with sustained failure timeout.
- get_
rng_ call_ count - Get the current RNG call count.
- has_
always_ violations - Check whether any always-type assertion was violated during this iteration.
- invariant_
fn - Create an invariant from a closure.
- method_
endpoint - Generate method endpoint from base endpoint and index.
- method_
uid - Generate UID for a specific interface method.
- panic_
on_ assertion_ violations - Panic if the report contains assertion violations.
- parse_
timeline - Parse a timeline string back into a recipe.
- reset_
always_ violations - Reset the violation counter. Must be called at the start of each iteration.
- reset_
assertion_ results - Reset all assertion statistics.
- reset_
rng_ call_ count - Reset the RNG call count to zero.
- reset_
sim_ rng - Reset the thread-local simulation RNG to a fresh state.
- sample_
duration - Sample a random duration from a range
- send
- Fire-and-forget delivery: send request unreliably with no reply.
- send_
request - Send a typed request and return a future for the response.
- serialize_
packet - Serialize a packet with token and payload.
- set_
rng_ breakpoints - Set RNG breakpoints for deterministic replay.
- set_
sim_ seed - Set the seed for the thread-local simulation RNG.
- sim_
random - Generate a random value using the thread-local simulation RNG.
- sim_
random_ range - Generate a random value within a specified range using the thread-local simulation RNG.
- sim_
random_ range_ or_ default - Generate a random value within the given range, returning the start value if the range is empty.
- try_
deserialize_ packet - Try to deserialize from a buffer that may contain partial data.
- try_
get_ reply - At-most-once delivery: send unreliably, race reply against disconnect.
- validate_
assertion_ contracts - Validate all assertion contracts based on their kind.
Type Aliases§
- Peer
Receiver - Type alias for the peer receiver channel.
Used when taking ownership via
take_receiver(). - Simulation
Result - A type alias for
Result<T, SimulationError>.
Attribute Macros§
- actor_
impl - Attribute macro for auto-generating
ActorHandlerboilerplate. - service
- Unified attribute macro for defining RPC interfaces and virtual actors.