Module simulated

Expand description

Simulate networking between peers with configurable link behavior (i.e. drops, latency, corruption, etc.).

Both peer and link modification can be performed dynamically over the lifetime of the simulated network. This can be used to mimic transient network partitions, offline nodes (that later connect), and/or degrading link quality. Messages on a link are delivered in order, and optional per-peer bandwidth limits account for transmission delay and queueing.

§Determinism

commonware-p2p::simulated can be run deterministically when paired with commonware-runtime::deterministic. This makes it possible to reproduce an arbitrary order of delivered/dropped messages with a given seed.

§Bandwidth Simulation

The simulator provides a realistic model of bandwidth contention where network capacity is a shared, finite resource. Bandwidth is allocated via progressive filling to provide max-min fairness.

If no bandwidth constraints are provided (default behavior), progressive filling and bandwidth tracking are not performed (avoiding unnecessary overhead for minimal p2p testing common in CI).

§Core Model

Whenever a transfer starts or finishes, or a bandwidth limit is updated, we execute a scheduling tick:

Collect Active Flows: Gather every active transfer that still has bytes to send. A flow is bound to one sender and to one receiver (if the message will be delivered).
Compute Progressive Filling: Run progressive filling to raise the rate of every active flow in lock-step until some sender’s egress or receiver’s ingress limit saturates (at which point the flow is frozen and the process repeats with what remains).
Wait for the Next Event: Using those rates, determine which flow will finish first by computing how long it needs to transmit its remaining bytes. Advance simulated time directly to that completion instant (advancing all other flows by the bytes transferred over the interval).
Deliver Message: Remove the completed flow and pass the message to the receiver. Repeat from step 1 until all flows are processed.

Messages between the same pair of peers remain strictly ordered. When one message finishes, the next message on that link may begin sending at arrival_time - new_latency so that its first byte arrives immediately after the previous one is fully received.

§Latency vs. Transmission Delay

The simulation correctly distinguishes between two key components of message delivery:

Transmission Delay: The time it takes to send all bytes of a message over the link. This is determined by the message size and the available bandwidth (e.g., a 10KB message on a 10KB/s link has a 1-second transmission delay).
Network Latency: The time it takes for a byte to travel from the sender to the receiver, independent of bandwidth. This is configured via the Link properties.

The final delivery time of a message is the sum of when its transmission completes plus the simulated network latency. This model ensures that large messages correctly occupy the network link for longer periods, affecting other concurrent transfers, while still accounting for the physical travel time of the data.

§Example

use commonware_p2p::{Manager, simulated::{Config, Link, Network}};
use commonware_cryptography::{ed25519, PrivateKey, Signer as _, PublicKey as _, };
use commonware_runtime::{deterministic, Metrics, Quota, Runner, Spawner};
use commonware_utils::NZU32;
use std::time::Duration;

// Generate peers
let peers = vec![
    ed25519::PrivateKey::from_seed(0).public_key(),
    ed25519::PrivateKey::from_seed(1).public_key(),
    ed25519::PrivateKey::from_seed(2).public_key(),
    ed25519::PrivateKey::from_seed(3).public_key(),
];

// Configure network
let p2p_cfg = Config {
    max_size: 1024 * 1024, // 1MB
    disconnect_on_block: true,
    tracked_peer_sets: Some(3),
};

// Rate limit quota (1000 messages per second per peer)
let quota = Quota::per_second(NZU32!(1000));

// Start context
let executor = deterministic::Runner::seeded(0);
executor.start(|context| async move {
    // Initialize network
    let (network, mut oracle) = Network::new(context.with_label("network"), p2p_cfg);

    // Start network
    let network_handler = network.start();

    // Register a peer set
    let mut manager = oracle.manager();
    manager.update(0, peers.clone().try_into().unwrap()).await;

    let (sender1, receiver1) = oracle.control(peers[0].clone()).register(0, quota).await.unwrap();
    let (sender2, receiver2) = oracle.control(peers[1].clone()).register(0, quota).await.unwrap();

    // Set bandwidth limits
    // peer[0]: 10KB/s egress, unlimited ingress
    // peer[1]: unlimited egress, 5KB/s ingress
    oracle.limit_bandwidth(peers[0].clone(), Some(10_000), None).await.unwrap();
    oracle.limit_bandwidth(peers[1].clone(), None, Some(5_000)).await.unwrap();

    // Link 2 peers
    oracle.add_link(
        peers[0].clone(),
        peers[1].clone(),
        Link {
            latency: Duration::from_millis(5),
            jitter: Duration::from_millis(2),
            success_rate: 0.75,
        },
    ).await.unwrap();

    // ... Use sender and receiver ...

    // Update link
    oracle.remove_link(
        peers[0].clone(),
        peers[1].clone(),
    ).await.unwrap();
    oracle.add_link(
        peers[0].clone(),
        peers[1].clone(),
        Link {
            latency: Duration::from_millis(100),
            jitter: Duration::from_millis(25),
            success_rate: 0.8,
        },
    ).await.unwrap();

    // ... Use sender and receiver ...

    // Shutdown network
    network_handler.abort();
});

Structs§

Config: Configuration for the simulated network.
ConnectedPeerProvider: Provides online peers from the simulated network.
Control: Individual control interface for a peer in the simulated network.
Link: Describes a connection between two peers.
Manager: Implementation of crate::Manager for peers.
Network: Implementation of a simulated network.
Oracle: Interface for modifying the simulated network.
Receiver: Implementation of a crate::Receiver for the simulated network.
Sender: Implementation of a crate::Sender for the simulated network.
SocketManager: Implementation of crate::Manager for peers with Addresses.
SplitSender: A sender that routes recipients per message via a user-provided function.
UnlimitedSender: Implementation of a crate::Sender for the simulated network without rate limiting.

Enums§

Error: Errors that can occur when interacting with the network.
SplitOrigin: Origin of a message in a split sender.
SplitTarget: Target for a message in a split receiver.

Traits§

SplitForwarder: A function that forwards messages from SplitOrigin to Recipients.
SplitRouter: A function that routes incoming Messages to a SplitTarget.