benchmark-rs
A benchmarking library for Rust package authors.
Benchmark-rs crate provides tools for Rust package authors to evaluate performance of their implementation for
varying workloads with varying configurations, find performance regressions between versions and also some crude
facilities to measure execution times of code blocks.
For example, if you implement a concurrent algorithm with an expectation of reduced execution time when more
CPU cores are added, benchmark-rs will help validate that the concurrency has the desired effect and that there
is no anomaly for varying data sets, that is, the algorithm performs consistently within boundaries defined for it.
However, this doesn't have to be a fancy algorithm that we want to test, some banal looking code can exhibit
radically different behavior for different data sets, depending on database queries, buffer sizes, number of
open files, etc. We may want to compare the behavior of our code with different querying strategies or different
buffer sizes for a range of data sets to find the best fit for our use-case.
Another common use-case is to verify that there is no regression in performance introduced by the new code. Benchmark-rs
supports comparison to previous results with user defined equality threshold.
Design
Benchmark-rs is designed to evaluate performance of subjects processing a set of workloads with modifiable configurations.
For example if we write a protocol parser we may define our workload points as 10MB, 20MB, ..., 100MB input sizes. Workloads
are user defined in benchmark-rs. Any type W: Clone + Display can be used to specify workloads. It could
be an integer that specifies the size of the workload, a path to a file or a key that we can use
to fetch the workload from the benchmark configuration. The Display is required
for the workload point to produce the key in result series, so it is recommended to keep it short and descriptive.
See examples below.
Each benchmark is repeated repeat times for every workload point after it was ramped up for that point. Upon
completion the summary is available as JSON or as CSV.
Issues
Issues are welcome and appreciated. Please submit to https://github.com/navigatorsguild/benchmark-rs/issues
Examples
A real world example can be found at
command-executor project
blocking_queue.rs
benchmark and at Benchmarks
wiki page which was built from the generated data.
Simple Benchmark
A simple benchmark that measures execution time for increasing workloads. In this case the workload is simulated by
by a u64 value passed to thread::sleep function
use std::thread;
use std::time::Duration;
use benchmark_rs::benchmarks::Benchmarks;
use benchmark_rs::stopwatch::StopWatch;
fn example(_stop_watch: &mut StopWatch, _config: &str, work: u64) -> Result<(), anyhow::Error> {
thread::sleep(Duration::from_millis(work));
Ok(())
}
fn main() -> Result<(), anyhow::Error> {
let mut benchmarks = Benchmarks::new("Example");
benchmarks.add("A Simple Benchmark", example, "No Configuration", (1..=10).collect(), 2, 1)?;
benchmarks.run()?;
let summary = benchmarks.summary_as_json();
println!("Summary: {summary}");
Ok(())
}
Benchmark Workloads
A more complex example that shows how to use Benchmark configuration and how to control the
stopwatch from within the benchmark to avoid measuring the housekeeping tasks.
use std::collections::BTreeMap;
use std::fmt::{Display, Formatter};
use std::thread;
use std::time::Duration;
use benchmark_rs::benchmarks::Benchmarks;
use benchmark_rs::stopwatch::StopWatch;
#[derive(Clone)]
struct Config {
pub resources: u32,
pub workloads: BTreeMap<u64, Duration>,
}
impl Display for Config {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
let keys: Vec<String> = self.workloads.keys().map(|k| k.to_string()).collect();
write!(f, "{}", keys.join(", "))
}
}
fn example(stop_watch: &mut StopWatch, config: Config, work: u64) -> Result<(), anyhow::Error> {
stop_watch.pause();
let sleep_time = config.workloads.get(&work).unwrap().clone();
stop_watch.resume();
thread::sleep(sleep_time / config.resources);
stop_watch.pause();
Ok(())
}
fn main() -> Result<(), anyhow::Error> {
let mut benchmarks = Benchmarks::new("benchmark-workloads");
let workloads: BTreeMap<u64, Duration> =
(0..=10).map(|i| (i, Duration::from_millis(i))).collect();
benchmarks.add(
"benchmark-workload-1",
example,
Config {
resources: 1,
workloads: workloads.clone(),
},
(1..=10).collect(),
2,
1,
)?;
benchmarks.add(
"benchmark-workload-2",
example,
Config {
resources: 2,
workloads: workloads.clone(),
},
(1..=10).collect(),
2,
1,
)?;
benchmarks.run()?;
let summary = benchmarks.summary_as_json();
println!("Benchmark summary in JSON format.");
println!("Summary:");
println!("{summary}");
println!();
println!("Benchmark series in CSV format.");
let csv_data = benchmarks.summary_as_csv(true, false);
for (k, v) in csv_data {
println!("Benchmark name: {k}");
for line in v {
println!("{line}")
}
println!();
}
Ok(())
}
{
"name": "benchmark-workloads",
"created_at": "2023-08-13 04:09:13.923036",
"series": {
"benchmark-workload-2": {
"name": "benchmark-workload-2",
"config": "0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10",
"runs": [
[
"1",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 639791,
"min_sec": 0.000639791,
"min_str": "00:00:00.000",
"max_nanos": 644250,
"max_sec": 0.00064425,
"max_str": "00:00:00.000",
"median_nanos": 642020,
"median_sec": 0.00064202,
"median_str": "00:00:00.000",
"std_dev": 3152.9891373108153,
"std_dev_sec": 3.1529891373108154e-6,
"std_dev_str": "00:00:00.000"
}
],
[
"2",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 1264124,
"min_sec": 0.001264124,
"min_str": "00:00:00.001",
"max_nanos": 1264250,
"max_sec": 0.00126425,
"max_str": "00:00:00.001",
"median_nanos": 1264187,
"median_sec": 0.001264187,
"median_str": "00:00:00.001",
"std_dev": 89.09545442950498,
"std_dev_sec": 8.909545442950498e-8,
"std_dev_str": "00:00:00.000"
}
],
[
"3",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 1891833,
"min_sec": 0.001891833,
"min_str": "00:00:00.001",
"max_nanos": 1892875,
"max_sec": 0.001892875,
"max_str": "00:00:00.001",
"median_nanos": 1892354,
"median_sec": 0.001892354,
"median_str": "00:00:00.001",
"std_dev": 736.8052659963826,
"std_dev_sec": 7.368052659963826e-7,
"std_dev_str": "00:00:00.000"
}
],
[
"4",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 2566417,
"min_sec": 0.002566417,
"min_str": "00:00:00.002",
"max_nanos": 2576416,
"max_sec": 0.002576416,
"max_str": "00:00:00.002",
"median_nanos": 2571416,
"median_sec": 0.002571416,
"median_str": "00:00:00.002",
"std_dev": 7070.360705084288,
"std_dev_sec": 7.0703607050842884e-6,
"std_dev_str": "00:00:00.000"
}
],
[
"5",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 2928666,
"min_sec": 0.002928666,
"min_str": "00:00:00.002",
"max_nanos": 3174917,
"max_sec": 0.003174917,
"max_str": "00:00:00.003",
"median_nanos": 3051791,
"median_sec": 0.003051791,
"median_str": "00:00:00.003",
"std_dev": 174125.75197396852,
"std_dev_sec": 0.00017412575197396852,
"std_dev_str": "00:00:00.000"
}
],
[
"6",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 3377791,
"min_sec": 0.003377791,
"min_str": "00:00:00.003",
"max_nanos": 3784625,
"max_sec": 0.003784625,
"max_str": "00:00:00.003",
"median_nanos": 3581208,
"median_sec": 0.003581208,
"median_str": "00:00:00.003",
"std_dev": 287675.0802172479,
"std_dev_sec": 0.00028767508021724787,
"std_dev_str": "00:00:00.000"
}
],
[
"7",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 4404499,
"min_sec": 0.004404499,
"min_str": "00:00:00.004",
"max_nanos": 4915541,
"max_sec": 0.004915541,
"max_str": "00:00:00.004",
"median_nanos": 4660020,
"median_sec": 0.00466002,
"median_str": "00:00:00.004",
"std_dev": 361361.26367113565,
"std_dev_sec": 0.00036136126367113564,
"std_dev_str": "00:00:00.000"
}
],
[
"8",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 4882584,
"min_sec": 0.004882584,
"min_str": "00:00:00.004",
"max_nanos": 5029209,
"max_sec": 0.005029209,
"max_str": "00:00:00.005",
"median_nanos": 4955896,
"median_sec": 0.004955896,
"median_str": "00:00:00.004",
"std_dev": 103679.53179147754,
"std_dev_sec": 0.00010367953179147753,
"std_dev_str": "00:00:00.000"
}
],
[
"9",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 5638959,
"min_sec": 0.005638959,
"min_str": "00:00:00.005",
"max_nanos": 5643416,
"max_sec": 0.005643416,
"max_str": "00:00:00.005",
"median_nanos": 5641187,
"median_sec": 0.005641187,
"median_str": "00:00:00.005",
"std_dev": 3151.5749237484424,
"std_dev_sec": 3.1515749237484423e-6,
"std_dev_str": "00:00:00.000"
}
],
[
"10",
{
"name": "benchmark-workload-2",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 5086250,
"min_sec": 0.00508625,
"min_str": "00:00:00.005",
"max_nanos": 5587292,
"max_sec": 0.005587292,
"max_str": "00:00:00.005",
"median_nanos": 5336771,
"median_sec": 0.005336771,
"median_str": "00:00:00.005",
"std_dev": 354290.19585927017,
"std_dev_sec": 0.00035429019585927015,
"std_dev_str": "00:00:00.000"
}
]
]
},
"benchmark-workload-1": {
"name": "benchmark-workload-1",
"config": "0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10",
"runs": [
[
"1",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 1259375,
"min_sec": 0.001259375,
"min_str": "00:00:00.001",
"max_nanos": 1276125,
"max_sec": 0.001276125,
"max_str": "00:00:00.001",
"median_nanos": 1267750,
"median_sec": 0.00126775,
"median_str": "00:00:00.001",
"std_dev": 11844.038584874672,
"std_dev_sec": 0.000011844038584874672,
"std_dev_str": "00:00:00.000"
}
],
[
"2",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 2513584,
"min_sec": 0.002513584,
"min_str": "00:00:00.002",
"max_nanos": 2520875,
"max_sec": 0.002520875,
"max_str": "00:00:00.002",
"median_nanos": 2517229,
"median_sec": 0.002517229,
"median_str": "00:00:00.002",
"std_dev": 5155.515541631118,
"std_dev_sec": 5.155515541631118e-6,
"std_dev_str": "00:00:00.000"
}
],
[
"3",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 3755750,
"min_sec": 0.00375575,
"min_str": "00:00:00.003",
"max_nanos": 3784000,
"max_sec": 0.003784,
"max_str": "00:00:00.003",
"median_nanos": 3769875,
"median_sec": 0.003769875,
"median_str": "00:00:00.003",
"std_dev": 19975.766568519968,
"std_dev_sec": 0.00001997576656851997,
"std_dev_str": "00:00:00.000"
}
],
[
"4",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 5003833,
"min_sec": 0.005003833,
"min_str": "00:00:00.005",
"max_nanos": 5030084,
"max_sec": 0.005030084,
"max_str": "00:00:00.005",
"median_nanos": 5016958,
"median_sec": 0.005016958,
"median_str": "00:00:00.005",
"std_dev": 18562.26011292806,
"std_dev_sec": 0.00001856226011292806,
"std_dev_str": "00:00:00.000"
}
],
[
"5",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 5518500,
"min_sec": 0.0055185,
"min_str": "00:00:00.005",
"max_nanos": 6283084,
"max_sec": 0.006283084,
"max_str": "00:00:00.006",
"median_nanos": 5900792,
"median_sec": 0.005900792,
"median_str": "00:00:00.005",
"std_dev": 540642.5311867353,
"std_dev_sec": 0.0005406425311867353,
"std_dev_str": "00:00:00.000"
}
],
[
"6",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 6942292,
"min_sec": 0.006942292,
"min_str": "00:00:00.006",
"max_nanos": 7541458,
"max_sec": 0.007541458,
"max_str": "00:00:00.007",
"median_nanos": 7241875,
"median_sec": 0.007241875,
"median_str": "00:00:00.007",
"std_dev": 423674.3416564189,
"std_dev_sec": 0.00042367434165641895,
"std_dev_str": "00:00:00.000"
}
],
[
"7",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 8784417,
"min_sec": 0.008784417,
"min_str": "00:00:00.008",
"max_nanos": 8812875,
"max_sec": 0.008812875,
"max_str": "00:00:00.008",
"median_nanos": 8798646,
"median_sec": 0.008798646,
"median_str": "00:00:00.008",
"std_dev": 20122.84477900677,
"std_dev_sec": 0.00002012284477900677,
"std_dev_str": "00:00:00.000"
}
],
[
"8",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 9426875,
"min_sec": 0.009426875,
"min_str": "00:00:00.009",
"max_nanos": 11596125,
"max_sec": 0.011596125,
"max_str": "00:00:00.011",
"median_nanos": 10511500,
"median_sec": 0.0105115,
"median_str": "00:00:00.010",
"std_dev": 1533891.3850889183,
"std_dev_sec": 0.0015338913850889183,
"std_dev_str": "00:00:00.001"
}
],
[
"9",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 9498333,
"min_sec": 0.009498333,
"min_str": "00:00:00.009",
"max_nanos": 9953333,
"max_sec": 0.009953333,
"max_str": "00:00:00.009",
"median_nanos": 9725833,
"median_sec": 0.009725833,
"median_str": "00:00:00.009",
"std_dev": 321733.5854398791,
"std_dev_sec": 0.0003217335854398791,
"std_dev_str": "00:00:00.000"
}
],
[
"10",
{
"name": "benchmark-workload-1",
"ramp_up": 1,
"repeat": 2,
"min_nanos": 11674000,
"min_sec": 0.011674,
"min_str": "00:00:00.011",
"max_nanos": 12627167,
"max_sec": 0.012627167,
"max_str": "00:00:00.012",
"median_nanos": 12150583,
"median_sec": 0.012150583,
"median_str": "00:00:00.012",
"std_dev": 673990.849303238,
"std_dev_sec": 0.0006739908493032379,
"std_dev_str": "00:00:00.000"
}
]
]
}
}
}
Benchmark name: benchmark-workload-2
point,ramp_up,repeat,min_sec,max_sec,median_sec,std_dev_sec
1,1,2,0.00063575,0.000637376,0.000636563,0.0000011497556262093261
2,1,2,0.001265333,0.001269584,0.001267458,0.0000030059109268240135
3,1,2,0.001890958,0.001892333,0.001891645,0.0000009722718241315028
4,1,2,0.002512042,0.002604791,0.002558416,0.0000655834468482711
5,1,2,0.003129084,0.00314525,0.003137167,0.000011431088224661727
6,1,2,0.003092583,0.003767833,0.003430208,0.0004774738539962162
7,1,2,0.004055333,0.004084166,0.004069749,0.000020388009821951726
8,1,2,0.004171542,0.005756541,0.004964041,0.0011207635410738967
9,1,2,0.004549334,0.005149792,0.004849563,0.0004245879236177119
10,1,2,0.006351,0.007404083,0.006877541,0.000744642130452273
Benchmark name: benchmark-workload-1
point,ramp_up,repeat,min_sec,max_sec,median_sec,std_dev_sec
1,1,2,0.001265417,0.001266709,0.001266063,0.0000009135819612930194
2,1,2,0.002518501,0.002534999,0.00252675,0.000011665847676015662
3,1,2,0.003762958,0.004072459,0.003917708,0.00021885025588401765
4,1,2,0.004471833,0.004559375,0.004515604,0.00006190154183863275
5,1,2,0.005676917,0.005765751,0.005721334,0.00006281512379992577
6,1,2,0.006610875,0.007539833,0.007075354,0.0006568725012374928
7,1,2,0.007831792,0.008021959,0.007926875,0.0001344683752579022
8,1,2,0.009668583,0.009832959,0.009750771,0.00011623138426431993
9,1,2,0.009677333,0.011178501,0.010427917,0.0010614860725002473
10,1,2,0.011440417,0.012526,0.011983208,0.0007676231008408358
use std::collections::BTreeMap;
use std::fmt::{Display, Formatter};
use std::thread;
use std::time::Duration;
use benchmark_rs::benchmarks::Benchmarks;
use benchmark_rs::stopwatch::StopWatch;
use rand::Rng;
#[derive(Clone)]
struct Config {
pub resources: u32,
pub workloads: BTreeMap<u64, Duration>,
}
impl Display for Config {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
for k in self.workloads.keys() {
write!(f, "{k} ")?;
}
Ok(())
}
}
fn example(_stop_watch: &mut StopWatch, config: Config, work: u64) -> Result<(), anyhow::Error> {
let sleep_time = config.workloads.get(&work).unwrap().clone();
thread::sleep(sleep_time / config.resources);
Ok(())
}
fn modified_example(
_stop_watch: &mut StopWatch,
config: Config,
work: u64,
) -> Result<(), anyhow::Error> {
let deviation = Duration::from_millis(rand::thread_rng().gen_range(0..10));
let sleep_time = config.workloads.get(&work).unwrap().clone();
thread::sleep(sleep_time / config.resources + deviation);
Ok(())
}
fn main() -> Result<(), anyhow::Error> {
let mut previous_benchmarks = Benchmarks::new("benchmarks");
let workloads: BTreeMap<u64, Duration> = (1..=10)
.map(|i| (i, Duration::from_millis(i * 25)))
.collect();
previous_benchmarks.add(
"benchmark-1",
example,
Config {
resources: 1,
workloads: workloads.clone(),
},
(1..=10).collect(),
5,
3,
)?;
previous_benchmarks.run()?;
let previous_summary = previous_benchmarks.summary_as_json();
let mut current_benchmarks = Benchmarks::new("benchmarks");
current_benchmarks.add(
"benchmark-1",
modified_example,
Config {
resources: 1,
workloads: workloads.clone(),
},
(1..=10).collect(),
5,
3,
)?;
current_benchmarks.add(
"benchmark-2",
modified_example,
Config {
resources: 2,
workloads: workloads.clone(),
},
(1..=10).collect(),
5,
3,
)?;
current_benchmarks.run()?;
let analysis_result = current_benchmarks.analyze(Some(previous_summary), 5.0)?;
println!("Analysis result:");
println!("{}", analysis_result.to_string());
assert!(!analysis_result.divergent_series().is_empty());
Ok(())
}
Analysis result. Current and previous values are in nanosecond units, the change is in percents.
{
"name": "benchmarks",
"new_series": [
"benchmark-2"
],
"equal_series": {},
"divergent_series": {
"benchmark-1": {
"8": {
"Equal": {
"point": "8",
"previous": 200686042,
"current": 206411375,
"change": 2.852880520709064
}
},
"7": {
"Equal": {
"point": "7",
"previous": 176700166,
"current": 183395333,
"change": 3.788998704166474
}
},
"10": {
"Equal": {
"point": "10",
"previous": 251701083,
"current": 256856000,
"change": 2.0480313149864315
}
},
"2": {
"Equal": {
"point": "2",
"previous": 52684667,
"current": 55052334,
"change": 4.494034288951653
}
},
"5": {
"Equal": {
"point": "5",
"previous": 127510583,
"current": 130817709,
"change": 2.5936090340046434
}
},
"6": {
"Equal": {
"point": "6",
"previous": 152803084,
"current": 158141750,
"change": 3.4938208446106955
}
},
"9": {
"Equal": {
"point": "9",
"previous": 225706250,
"current": 229522083,
"change": 1.6906191122310474
}
},
"1": {
"Greater": {
"point": "1",
"previous": 26413209,
"current": 35264166,
"change": 33.50958605597674
}
},
"4": {
"Equal": {
"point": "4",
"previous": 104201208,
"current": 109166958,
"change": 4.7655397622645665
}
},
"3": {
"Equal": {
"point": "3",
"previous": 79801875,
"current": 84060834,
"change": 5.336915955922095
}
}
}
}
}
Similar Projects
License: MIT OR Apache-2.0
