Crate balter

source ·
Expand description

§Balter

Linux build status Crates.io

Balter, short for Build A Load TestER, is a load/stress testing framework for Rust designed to be flexible, efficient, and simple to use. Balter aims to minimize the conceptual overhead of load testing, and builds off of Tokio and the async ecosystem.

§Features

  • Write load tests using normal Rust code
  • Run scenarios with a given transaction per second (TPS)
  • Run scenarios to find the max TPS at a given error rate
  • Run scenarios distributed across multiple machines

§WIP

  • More robust logic
  • mTLS
  • Peer discovery via DNS
  • Autoscaling hooks
  • More customizability (eg. keyed transaction limits)
  • Efficiency improvements

§How To Use

Balter is designed to be simple to get started with while allowing for advanced workflows. At the core of the abstraction are two concepts: (1) the transaction and (2) the scenario.

First you need to include Balter in your Cargo.toml.

[dependencies]
balter = "0.1"

A transaction is a single request to your service. A transaction is the way Balter measures the number and timing of requests being made to the service, as well as the error rate and success rate. These are used to make various scaling decisions. You denote a transaction with the #[transaction] macro. Currently Balter only supports transactions which are async functions with any number of arguments that return a Result<T, E>, though this will be made more flexible in the future (such as supporting Option<T> or infallible transactions).

ⓘ
use balter::prelude::*;

#[transaction]
async fn my_transaction(foo: Foo) -> Result<Bar, MyError> {
    ...
}

#[transaction]
async fn other_transaction(foo: Foo, foo2: Foo2, ...) -> Result<Bar, MyError> {
    ...
}

A scenario is a function which calls any number of transactions (or other scenarios). Similar to transactions, a scenario is denoted with the #[scenario] macro. By default, scenarios behave identically to normal async Rust functions.

ⓘ
use balter::prelude::*;

#[scenario]
async fn my_scenario() {
    let _ = my_transaction().await;
}

#[scenario]
async fn my_other_scenario() {
    let res = my_transaction().await;

    for i in res.count {
        other_transaction().await;
    }
}

#[scenario]
async fn my_nested_scenario() {
    my_scenario().await;
    my_other_scenario().await;
}

What makes scenarios different from regular async functions is that they have additional methods allowing you to specify load testing functionality.

  • .tps(u32) Run a scenario at a specified TPS
  • .saturate() Run a scenario, increasing the TPS until a 3% error rate
  • .overload() Run a scenario, increasing the TPS until an 80% error rate
  • .error_rate(f64) Run a scenario, increasing the TPS until a custom error rate
ⓘ
// Run scenario at 300 TPS for 30 seconds
my_scenario().tps(300u32).duration(Duration::from_secs(30)).await;

// Increase TPS until we see an error rate of 3% for 120 seconds
my_scenario().saturate().duration(Duration::from_secs(120)).await;

// Increase TPS until we see an error rate of 80% for 120 seconds
my_scenario().overload().duration(Duration::from_secs(120)).await;

// Run a scenario increasing the TPS until a specified error rate:
// Increase TPS until we see an error rate of 25% for 120 seconds
my_scenario().error_rate(0.25).duration(Duration::from_secs(120)).await;

You can run scenarios together for more complicated load test scenarios using standard tokio async code:

ⓘ
use balter::prelude::*;

#[scenario]
async fn my_root_scenario() {
    // In series
    my_scenario_a().tps(300).duration(Duration::from_secs(120));
    my_scenario_b().saturate().duration(Duration::from_secs(120));

    // In parallel
    tokio::join! {
        my_scenario_a().tps(300).duration(Duration::from_secs(120)),
        my_scenario_b().saturate().duration(Duration::from_secs(120)),
    }
}

Currently Balter only supports scenarios which are async functions which take no arguments and return no values; this unfortunately limits them a bit right now but is a restriction which will be lifted soon. Additionally, scenarios must supply a duration, but this is also a restriction which will be lifted soon (and will result in a scenario which runs indefinitely).

All put together, a simple single-server load test looks like the following:

use balter::prelude::*;
use std::time::Duration;

#[tokio::main]
async fn main() {
    my_scenario()
        .tps(500)
        .duration(Duration::from_secs(30))
        .await;

    my_scenario()
        .saturate()
        .duration(Duration::from_secs(120))
        .await;
}

#[scenario]
async fn my_scenario() {
    my_transaction().await;
}

#[transaction]
async fn my_transaction() -> Result<u32, String> {
    // Some request logic...

    Ok(0)
}

§Distributed Support (Experimental)

Running a load test on a single server is limited, and Balter aims to provide a distributed runtime. Currently Balter supports distributed load tests, but they are fragile and not efficient. This functionality will improve over time, but the current support should be considered experimental.

To use the distributed runtime, you need to set the rt feature flag. You will also need to add linkme to your dependencies list.

[dependencies]
balter = { version = "0.1", features = ["rt"] }
linkme = "0.3"

The next step is to instantiate the runtime. This is needed in order to set up the server and gossip functionality.

use balter::prelude::*;

#[tokio::main]
async fn main() {
    BalterRuntime::new().with_args().run().await;
}

Note that we call .with_args() on the runtime. This sets up the binary to accept CLI arguments for the port (-p) and for peer addresses (-n). You can also use the builder pattern with .port() and .peers(), which are documented in the rustdocs. In order to have distributed load testing support, each instantiation of the service needs to know of the address of at least one peer, otherwise the gossip functionality won’t work. Support will be added for DNS support to allow for more dynamic addresses. With the runtime configured, you can spin up the servers.

Assuming the first server is running on 127.0.0.1:7621 (the first server does not need any peer addresses), each subsequent service can be started like so:

$ ./load_test_binary -n 127.0.0.1:7621

Once the services are all pointed at each other, they will begin to gossip and coordinate. To start a load test, you make an HTTP request to the /run endpoint of any of the services in the mesh with the name being the function name of the scenario you would like to run. Currently you must specify all parameters, and .saturate(), overload() and error_rate() are all under one header (Saturate).

$ # For running a TPS load test
$ curl "127.0.0.1:7621/run" --json '{ "name": "my_scenario", "duration": 30, "kind": { "Tps": 500 }}'

$ # For running a saturate/overload/error_rate load test (`.saturate()` is 0.03, `.overload()` is 0.80)
$ curl "127.0.0.1:7621/run" --json '{ "name": "my_scenario", "duration": 30, "kind": { "Saturate": 0.03 }}'

§Limitations

Balter is a Beta framework and there are some rough edges. This section lists the most important to be aware of. All of these are limitations being worked on, as the goal of Balter is to be as flexible as is needed.

  • Various type restrictions

    • Scenario must be functions which take no arguments and return no values
    • Transactions must be functions which return a Result<T, E>

  • The current statistical engine powering the TPS and Saturate functionality is not perfect

    • It cannot handle changing loads well. It finds a set point and then sits there.
    • A highly variable measurement may cause it to find a bad set point.

  • The distributed functionality is experimental.

    • Inefficient Gossip protocol being used
    • No transaction security (no TLS, mTLS, or WSS) - use at your own risk (and in a private VPC)
    • Likely to run into weird error cases

§How It Works

Balter works by continuously measuring transaction timings and transaction success rates. It does this via tokio::task_local!: the scenarios create hooks via the task_local! which the transactions submit data to. Balter uses this data to scale up the TPS on the machine a scenario started on by increasing the number of concurrent tasks.

If the TPS required for a scenario is too high for the given server to handle, it will find the optimal parallel task count to maximize the output TPS of itself. If the distributed runtime is being used, then the server requests help from its peers.

The distributed runtime has two primary tasks being run in the background: (1) the API server and (2) the gossip protocol. The API server is to handle the initial /run request, as well as for setting up the websocket support for the gossip protocol.

The gossip protocol is run over websockets, and is fairly crude at this point. The only state shared about each peer is whether it is free or busy (or down). A peer is only busy if it has asked other servers for help. Consensus is done by sending all information between two nodes and each taking the max of the intersection, given that the data is monotonic.

§Developer Notes

The Balter repository is set up to be easy to get started with development. It uses Nix to facilitate the environment setup via shell.nix (if you haven’t yet drank the Nixaide, open up that file and it will give you an idea of the programs you’ll want). The two most important for testing are just and cpulimit which are both used for running the test environment. (Just simplifies running the test scripts, but isn’t actually needed)

Balter currently has most of its testing as integration tests, run via just {test name} which calls the test scripts in test-scripts/. Unfortunately, cpulimit is going to be based on the computer you’re running on, so it might require tweaking some values in the test-scripts/. For a basic test to get you started, just basic-saturate will do.

The various integration test scripts all start up a mock-service (code in mock-service/) which is a simple HTTP server with a few endpoints to make it easier to test various functionality. /api_10ms is an endpoint which takes 10ms to respond. /api_max_tps is an endpoint which has a set MAX_TPS it accepts before responding with errors (and also coincidentally takes 10ms to respond).

Re-exports§

Modules§

Attribute Macros§