simplebench-runtime 2.0.0

Core runtime library for SimpleBench - a minimalist microbenchmarking framework
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153

# SimpleBench

[![Crates.io](https://img.shields.io/crates/v/cargo-simplebench.svg)](https://crates.io/crates/cargo-simplebench)
[![Documentation](https://docs.rs/simplebench-runtime/badge.svg)](https://docs.rs/simplebench-runtime)

A minimalist microbenchmarking framework for Rust with clear regression detection.

SimpleBench provides a simple `#[bench]` attribute and automatic workspace-wide benchmark discovery, without the complexity of larger frameworks.

## Features

- **Simple API** - Just add `#[bench]` to any function
- **Workspace support** - Automatically discovers and runs benchmarks across all crates
- **Regression detection** - Statistical comparison against baselines with configurable thresholds
- **Low variance** - CPU pinning, warmup phases, and high sample counts for reliable measurements
- **CI-ready** - `--ci` flag returns non-zero exit code on regressions
- **Historical tracking** - Stores run history for trend analysis

## Quick Start

### Installation

```bash
cargo install cargo-simplebench
```

### Add Dependencies

In your crate's `Cargo.toml`:

```toml
[dev-dependencies]
simplebench-runtime = "1.0.6"
simplebench-macros = "1.0.6"
```

### Write a Benchmark

```rust
#[cfg(test)]
mod benchmarks {
    use simplebench_macros::bench;

    #[bench]
    fn my_function() {
        // Code to benchmark
        let sum: u64 = (0..1000).sum();
        std::hint::black_box(sum);
    }
}
```

### Run Benchmarks

```bash
cargo simplebench
```

## Configuration

### Command Line Options

```bash
cargo simplebench [OPTIONS]

Options:
  --samples <N>           Number of samples per benchmark (default: 1000)
  --iterations <N>        Iterations per sample (default: 1000)
  --warmup-duration <S>   Warmup duration in seconds (default: 5)
  --threshold <P>         Regression threshold percentage (default: 5.0)
  --ci                    CI mode - exit with error on regression
  --filter <PATTERN>      Run only benchmarks matching pattern
  --parallel              Run benchmarks in parallel (faster, may increase variance)
```

### Environment Variables

All options can also be set via environment variables:

- `SIMPLEBENCH_SAMPLES`
- `SIMPLEBENCH_ITERATIONS`
- `SIMPLEBENCH_WARMUP_DURATION`
- `SIMPLEBENCH_THRESHOLD`
- `SIMPLEBENCH_BENCH_FILTER`

### Configuration File

Create `.simplebench.toml` in your project root:

```toml
[measurement]
samples = 1000
iterations = 1000
warmup_duration_secs = 5

[comparison]
threshold = 5.0
```

## CI Integration

```yaml
# .github/workflows/bench.yml
name: Benchmarks
on: [push, pull_request]

jobs:
  bench:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - run: cargo install cargo-simplebench
      - run: cargo simplebench --ci
```

## Subcommands

```bash
# Run benchmarks
cargo simplebench

# Clean baseline data
cargo simplebench clean

# Analyze historical trends
cargo simplebench analyze <benchmark_name> --last 10
```

## How It Works

SimpleBench uses the `inventory` crate for compile-time benchmark registration. The `#[bench]` macro expands to register each benchmark function, and `cargo simplebench` builds a unified runner that links all workspace crates and executes discovered benchmarks.

Benchmarks are compiled with `#[cfg(test)]`, so they're excluded from production builds.

## Crates

- [`cargo-simplebench`](https://crates.io/crates/cargo-simplebench) - CLI tool
- [`simplebench-runtime`](https://crates.io/crates/simplebench-runtime) - Core runtime library
- [`simplebench-macros`](https://crates.io/crates/simplebench-macros) - `#[bench]` proc-macro

## License

Licensed under either of:

- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

---

*This project was co-authored with [Claude](https://claude.ai), an AI assistant by Anthropic.*