Crate turmoil

source ·
Expand description

Turmoil is a framework for testing distributed systems. It provides deterministic execution by running multiple concurrent hosts within a single thread. It introduces “hardship” into the system via changes in the simulated network. The network can be controlled manually or with a seeded rng.

§Hosts and Software

A turmoil simulation is comprised of one or more hosts. Hosts run software, which is represented as a Future.

Test code is also executed on a special host, called a client. This allows for an entry point into the simulated system. Client hosts have all the same capabilities as normal hosts, such as networking support.

use turmoil;

let mut sim = turmoil::Builder::new().build();

// register a host
sim.host("host", || async {

    // host software goes here

    Ok(())
});

// define test code
sim.client("test", async {

    // we can interact with other hosts from here

    Ok(())
});

// run the simulation and handle the result
_ = sim.run();

§Networking

Simulated networking types that mirror tokio::net are included in the turmoil::net module.

Turmoil is not yet oppinionated on how to structure your application code to swap in simulated types under test. More on this coming soon…

§Network Manipulation

The simulation has the following network manipulation capabilities:

  • partition, which introduces a network partition between hosts
  • repair, which repairs a network partition between hosts
  • hold, which holds all “in flight” messages between hosts. Messages are available for introspection using Sim’s links method.
  • release, which releases all “in flight” messages between hosts

§Tracing

The tracing crate is used to emit important events during the lifetime of a simulation. To enable traces, your tests must install a tracing-subscriber. The log level of turmoil can be configured using RUST_LOG=turmoil=info.

It is possible to configure your tracing subscriber to log elapsed simulation time instead of real time. See the grpc example.

Turmoil can provide a full packet level trace of the events happening in a simulation by passing RUST_LOG=turmoil=trace. This is really useful when you are unable to identify why some unexpected behaviour is happening and you need to know which packets are reaching where.

To see this in effect, you can run the axum example with the following command:

RUST_LOG=INFO,turmoil=TRACE cargo run -p axum-example

You can see the TCP packets being sent and delivered between the server and the client:

...
2023-11-29T20:23:43.276745Z TRACE node{name="server"}: turmoil: Send src=192.168.0.1:9999 dst=192.168.0.2:49152 protocol=TCP [0x48, 0x54, 0x54, 0x50, 0x2F, 0x31, 0x2E, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4F, 0x4B, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x74, 0x79, 0x70, 0x65, 0x3A, 0x20, 0x74, 0x65, 0x78, 0x74, 0x2F, 0x70, 0x6C, 0x61, 0x69, 0x6E, 0x3B, 0x20, 0x63, 0x68, 0x61, 0x72, 0x73, 0x65, 0x74, 0x3D, 0x75, 0x74, 0x66, 0x2D, 0x38, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x3A, 0x20, 0x31, 0x30, 0xD, 0xA, 0x64, 0x61, 0x74, 0x65, 0x3A, 0x20, 0x57, 0x65, 0x64, 0x2C, 0x20, 0x32, 0x39, 0x20, 0x4E, 0x6F, 0x76, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x32, 0x30, 0x3A, 0x32, 0x33, 0x3A, 0x34, 0x33, 0x20, 0x47, 0x4D, 0x54, 0xD, 0xA, 0xD, 0xA, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x66, 0x6F, 0x6F, 0x21]
2023-11-29T20:23:43.276834Z DEBUG turmoil::sim: step 43
2023-11-29T20:23:43.276907Z DEBUG turmoil::sim: step 44
2023-11-29T20:23:43.276981Z DEBUG turmoil::sim: step 45
2023-11-29T20:23:43.277039Z TRACE node{name="client"}: turmoil: Delivered src=192.168.0.1:9999 dst=192.168.0.2:49152 protocol=TCP [0x48, 0x54, 0x54, 0x50, 0x2F, 0x31, 0x2E, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4F, 0x4B, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x74, 0x79, 0x70, 0x65, 0x3A, 0x20, 0x74, 0x65, 0x78, 0x74, 0x2F, 0x70, 0x6C, 0x61, 0x69, 0x6E, 0x3B, 0x20, 0x63, 0x68, 0x61, 0x72, 0x73, 0x65, 0x74, 0x3D, 0x75, 0x74, 0x66, 0x2D, 0x38, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x3A, 0x20, 0x31, 0x30, 0xD, 0xA, 0x64, 0x61, 0x74, 0x65, 0x3A, 0x20, 0x57, 0x65, 0x64, 0x2C, 0x20, 0x32, 0x39, 0x20, 0x4E, 0x6F, 0x76, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x32, 0x30, 0x3A, 0x32, 0x33, 0x3A, 0x34, 0x33, 0x20, 0x47, 0x4D, 0x54, 0xD, 0xA, 0xD, 0xA, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x66, 0x6F, 0x6F, 0x21]
2023-11-29T20:23:43.277097Z TRACE node{name="client"}: turmoil: Recv src=192.168.0.1:9999 dst=192.168.0.2:49152 protocol=TCP [0x48, 0x54, 0x54, 0x50, 0x2F, 0x31, 0x2E, 0x31, 0x20, 0x32, 0x30, 0x30, 0x20, 0x4F, 0x4B, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x74, 0x79, 0x70, 0x65, 0x3A, 0x20, 0x74, 0x65, 0x78, 0x74, 0x2F, 0x70, 0x6C, 0x61, 0x69, 0x6E, 0x3B, 0x20, 0x63, 0x68, 0x61, 0x72, 0x73, 0x65, 0x74, 0x3D, 0x75, 0x74, 0x66, 0x2D, 0x38, 0xD, 0xA, 0x63, 0x6F, 0x6E, 0x74, 0x65, 0x6E, 0x74, 0x2D, 0x6C, 0x65, 0x6E, 0x67, 0x74, 0x68, 0x3A, 0x20, 0x31, 0x30, 0xD, 0xA, 0x64, 0x61, 0x74, 0x65, 0x3A, 0x20, 0x57, 0x65, 0x64, 0x2C, 0x20, 0x32, 0x39, 0x20, 0x4E, 0x6F, 0x76, 0x20, 0x32, 0x30, 0x32, 0x33, 0x20, 0x32, 0x30, 0x3A, 0x32, 0x33, 0x3A, 0x34, 0x33, 0x20, 0x47, 0x4D, 0x54, 0xD, 0xA, 0xD, 0xA, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x66, 0x6F, 0x6F, 0x21]
2023-11-29T20:23:43.277324Z  INFO client: axum_example: Got response: Response { status: 200, version: HTTP/1.1, headers: {"content-type": "text/plain; charset=utf-8", "content-length": "10", "date": "Wed, 29 Nov 2023 20:23:43 GMT"}, body: b"Hello foo!" }
...

Here the server is sending a response, before it is delivered to, and received by the client. Note that there are three steps to each packet trace in turmoil. We see Send when a packet is sent from one address to another. The packet is then Delivered to its destination, and when the destination reads the packet it is Recv’d.

§Feature flags

  • regex: Enables regex host resolution through ToIpAddrs

§tokio_unstable

Turmoil uses unhandled_panic to forward host panics as test failures. See unstable features to opt in.

Modules§

  • net
    This module contains the simulated TCP/UDP networking types.

Structs§

  • Builder
    A builder that can be used to configure the simulation.
  • Datagram
    UDP datagram.
  • LinkIter
    An iterator for the link, providing access to sent messages that have not yet been delivered.
  • LinksIter
    An iterator for the network topology, providing access to all active links in the simulated network.
  • SentRef
    Provides a reference to a message that is currently inflight on the network from one host to another.
  • Sim
    A handle for interacting with the simulation.

Enums§

  • IpVersion
    The kinds of networks that can be simulated in turmoil
  • Protocol
    Supported network protocols.
  • Segment
    This is a simplification of real TCP.

Traits§

Functions§

  • elapsed
    Returns how long the currently executing host has been executing for in virtual time.
  • hold
    Hold messages between two hosts, or sets of hosts, until release is called.
  • lookup
    Lookup an IP address by host name.
  • lookup_many
    Lookup an IP address by host name. Use regex to match a number of hosts.
  • partition
    Partition two hosts, or sets of hosts, resulting in all messages sent between them to be dropped.
  • release
    The opposite of hold. All held messages are immediately delivered.
  • repair
    Repair the connection between two hosts, or sets of hosts, resulting in messages to be delivered.
  • reverse_lookup
    Perform a reverse DNS lookup, returning the hostname if the entry exists.
  • sim_elapsed
    Returns how long the simulation has been executing for in virtual time.

Type Aliases§