yab 0.2.0

Yet Another Benchmarking framework powered by `cachegrind`
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163

# Yet Another Benchmarking framework powered by cachegrind


[![CI](https://github.com/slowli/yab/actions/workflows/ci.yml/badge.svg)](https://github.com/slowli/yab/actions/workflows/ci.yml)
[![License: MIT OR Apache-2.0](https://img.shields.io/badge/License-MIT%2FApache--2.0-blue)](https://github.com/slowli/yab#license)
![rust 1.86+ required](https://img.shields.io/badge/rust-1.86+-blue.svg?label=Required%20Rust)

**Documentation:** [![Docs.rs](https://img.shields.io/docsrs/yab.svg)](https://docs.rs/yab/)
[![crate docs (main)](https://img.shields.io/badge/main-yellow.svg?label=docs)](https://slowli.github.io/yab/yab/)

YAB is **Y**et **A**nother **B**enchmarking framework powered by [`cachegrind`] from the Valgrind tool suite.
It collects reproducible measurements of Rust code (e.g., the number of executed instructions,
number of L1 and L2/L3 cache hits and RAM accesses), making it possible to use in CI etc.

## Features


- Supports newer `cachegrind` versions and customizing the `cachegrind` wrapper.
- Supports capturing only instruction counts (i.e., not simulating CPU caches).
- Conditionally injects `CACHEGRIND_{START|STOP}_INSTRUMENTATION` macros (available in `cachegrind`
  3.22.0+) allowing for more precise measurements.
- Supports configurable warm-up (defined in terms of executed instructions) before the capture.

## Usage


Define a benchmark binary and include it into your crate manifest:

```toml
[dev-dependencies]
yab = "0.2.0"

[[bench]]
name = "your_bench"
harness = false
```

In the bench source (`benches/your_bench.rs` in the example above), define a function with signature `fn(&mut yab::Bencher)`
and wrap it in the `yab::main!` macro:

```rust
use yab::Bencher;

fn benchmarks(bencher: &mut Bencher) {
    // define your benchmarking code here
}

yab::main!(benchmarks);
```

Run benchmarks as usual using `cargo bench` (or `cargo test --bench ...` to test them).

### Configuration options


Run `cargo bench ... -- --help` to get help on the supported configuration options. Some of the
common options are:

- `--list`: lists benchmarks without running them.
- `--print`: prints results of the latest run instead of running benchmarks.
- `--jobs N` / `-j N`: specifies the number of benchmarks to run in parallel. By default, it's equal
  to the number of logical CPUs in the system.
- `--verbose`, `--quiet`: increases or decreases verbosity of benchmarking output.

### Examples


```rust
use yab::{black_box, Bencher, BenchmarkId};

/// Suppose we want to benchmark this function
fn fibonacci(n: u64) -> u64 {
    match n {
        0 | 1 => 1,
        n => fibonacci(n - 1) + fibonacci(n - 2),
    }
}

fn benchmarks(bencher: &mut Bencher) {
    // Benchmark simple functions.
    bencher
        .bench("fib_short", || fibonacci(black_box(10)))
        .bench("fib_long", || fibonacci(black_box(30)));
    // It's possible to benchmark parametric functions as well:
    for n in [15, 20, 25] {
        bencher.bench(
            BenchmarkId::new("fib", n),
            || fibonacci(black_box(n)),
        );
    }
    // To account for setup and/or teardown, you may use `bench_with_capture`
    bencher.bench_with_capture("fib_capture", |capture| {
        // This will not be included into captured stats.
        black_box(fibonacci(black_box(30)));
        // This will be the only captured segment.
        let output = capture.measure(|| fibonacci(black_box(10)));
        // This assertion won't be captured either
        assert_eq!(output, 55);
    });
}

yab::main!(benchmarks);
```

Here's sample benchmark output:

![Basic benchmark output](examples/basic.svg)

More verbose output with `--verbose` option also showcasing changes to the benchmarked function:

![Verbose benchmark output](examples/verbose.svg)

Verbose output also outputs breakdown by function:

![Breakdown by function](examples/breakdown.svg)

### Baselines


Similar to [`criterion`], `yab` allows managing named *baselines* for benchmarks.

- To save a baseline, specify its name via the `--save-baseline` (or `--save`) argument.
- To compare against a previously saved baseline, specify its name with the `--baseline` (or `--vs`) arg.
- To print the baseline data, specify its name with the `--print` arg.

By default, baselines are stored inside the `target/yab` directory like the other collected data.
However, if the baseline name is prefixed with `pub:` (short for "public"),
it's located in the `benches/$bench_crate_name` directory (i.e., near the bench code).
This allows easily checking baselines into git to be used in CI etc.

![Comparing to baseline](examples/cmp-baseline.svg)

If the baseline doesn't contain any benchmarks from the code, it'll be highlighted as warnings:

![Printing baseline data](examples/print-baseline.svg)

When comparing against a baseline, the benchmark will fail on significant regression.
Regression is defined in terms of executed instructions. The regression threshold is 5% by default;
it can be controlled via the `--threshold` command-line arg or the `CACHEGRIND_REGRESSION_THRESHOLD` env var.

![Benchmark failing on regression](examples/baseline-regression.svg)

## Limitations


- `cachegrind` has somewhat limited platform support (e.g., doesn't support Windows).
- `cachegrind` uses simplistic / outdated CPU cache simulation to the point that recent versions
  disable this simulation altogether by default.
- `cachegrind` has limited support when simulating multi-threaded environment.
- Even small changes in the benchmarked code can lead to (generally small) divergences in the measured stats.

## Alternatives and similar tools


- This crate is heavily inspired by [`iai`](https://crates.io/crates/iai), *the* original `cachegrind`-based
  benchmarking framework for Rust.
- [`iai-callgrind`](https://crates.io/crates/iai-callgrind) is an extended / reworked fork of `iai`.
  Compared to it, `yab` prefers simplicity to versatility.
- Benchmarking APIs are inspired by [`criterion`].

## License


Licensed under either of [Apache License, Version 2.0](LICENSE-APACHE)
or [MIT license](LICENSE-MIT) at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in `yab` by you, as defined in the Apache-2.0 license,
shall be dual licensed as above, without any additional terms or conditions.

[`cachegrind`]: https://valgrind.org/docs/manual/cg-manual.html
[`criterion`]: https://crates.io/crates/criterion