Expand description
Lightweight benchmarking toolkit focused on practical performance analysis and report generation.
§benchkit
Practical, Documentation-First Benchmarking for Rust.
benchkit
is a lightweight toolkit for performance analysis, born from the hard-learned lessons of optimizing high-performance libraries. It rejects rigid, all-or-nothing frameworks in favor of flexible, composable tools that integrate seamlessly into your existing workflow.
§The Benchmarking Dilemma
In Rust, developers often face a frustrating choice:
- The Heavy Framework (
criterion
): Statistically powerful, but forces a rigid structure (benches/
), complex setup, and produces reports that are difficult to integrate into your project’s documentation. You must adapt your project to the framework. - The Manual Approach (
std::time
): Simple to start, but statistically naive. It leads to boilerplate, inconsistent measurements, and conclusions that are easily skewed by system noise.
benchkit
offers a third way.
§A Toolkit, Not a Framework
This is the core philosophy of benchkit
. It doesn’t impose a workflow; it provides a set of professional, composable tools that you can use however you see fit.
- âś… Integrate Anywhere: Write benchmarks in your test files, examples, or binaries. No required directory structure.
- âś… Documentation-First: Treat performance reports as a first-class part of your documentation, with tools to automatically keep them in sync with your code.
- âś… Practical Focus: Surface the key metrics needed for optimization decisions, hiding deep statistical complexity until you ask for it.
- âś… Zero Setup: Start measuring performance in minutes with a simple, intuitive API.
§🚀 Quick Start: Compare, Analyze, and Document
This example demonstrates the core benchkit
workflow: comparing two algorithms and automatically updating a performance section in your readme.md
.
1. Add to dev-dependencies
in Cargo.toml
:
[dev-dependencies]
benchkit = { version = "0.1", features = [ "full" ] }
2. Create a benchmark in your tests
directory:
// In tests/performance_test.rs
#![ cfg( feature = "integration" ) ]
use benchkit::prelude::*;
fn generate_data( size : usize ) -> Vec< u32 >
{
( 0..size ).map( | x | x as u32 ).collect()
}
#[ test ]
fn update_readme_performance_docs()
{
let mut comparison = ComparativeAnalysis::new( "Sorting Algorithms" );
let data = generate_data( 1000 );
// Benchmark the first algorithm
comparison = comparison.algorithm
(
"std_stable_sort",
{
let mut d = data.clone();
move ||
{
d.sort();
}
}
);
// Benchmark the second algorithm
comparison = comparison.algorithm
(
"std_unstable_sort",
{
let mut d = data.clone();
move ||
{
d.sort_unstable();
}
}
);
// Run the comparison and update the documentation
let report = comparison.run();
let markdown = report.to_markdown();
let updater = MarkdownUpdater::new( "readme.md", "Performance" );
updater.update_section( &markdown ).unwrap();
}
3. Add a placeholder section to your readme.md
:
## Performance
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
### Key Insights
- **Fastest operation**: create_user (36.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
### Key Insights
- **Fastest operation**: get_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |
| create_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |
### Key Insights
- **Fastest operation**: get_user (64.00ns)
- **Performance range**: 1.0x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
### Key Insights
- **Fastest operation**: create_user (40.00ns)
- **Performance range**: 1.0x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 24.00ns | 41666667 | 0.00ns | 40.00ns | 21.00ns |
| create_user | 28.00ns | 35714286 | 0.00ns | 40.00ns | 19.00ns |
### Key Insights
- **Fastest operation**: get_user (24.00ns)
- **Performance range**: 1.2x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| get_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
### Key Insights
- **Fastest operation**: create_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| get_user | 88.00ns | 11363636 | 80.00ns | 120.00ns | 17.00ns |
### Key Insights
- **Fastest operation**: create_user (84.00ns)
- **Performance range**: 1.0x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| create_user | 92.00ns | 10869565 | 80.00ns | 120.00ns | 19.00ns |
### Key Insights
- **Fastest operation**: get_user (84.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## Performance
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
### Key Insights
- **Fastest operation**: create_user (36.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
### Key Insights
- **Fastest operation**: get_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |
| create_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |
### Key Insights
- **Fastest operation**: get_user (64.00ns)
- **Performance range**: 1.0x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
### Key Insights
- **Fastest operation**: create_user (40.00ns)
- **Performance range**: 1.0x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 24.00ns | 41666667 | 0.00ns | 40.00ns | 21.00ns |
| create_user | 28.00ns | 35714286 | 0.00ns | 40.00ns | 19.00ns |
### Key Insights
- **Fastest operation**: get_user (24.00ns)
- **Performance range**: 1.2x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| get_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
### Key Insights
- **Fastest operation**: create_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| get_user | 88.00ns | 11363636 | 80.00ns | 120.00ns | 17.00ns |
### Key Insights
- **Fastest operation**: create_user (84.00ns)
- **Performance range**: 1.0x difference between fastest and slowest
## api_performance Results
| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| create_user | 92.00ns | 10869565 | 80.00ns | 120.00ns | 19.00ns |
### Key Insights
- **Fastest operation**: get_user (84.00ns)
- **Performance range**: 1.1x difference between fastest and slowest
## đź§° What's in the Toolkit?
`benchkit` provides a suite of composable tools. Use only what you need.
<details>
<summary><strong>Measure: Core Timing and Profiling</strong></summary>
At its heart, `benchkit` provides simple and accurate measurement primitives.
```rust
use benchkit::prelude::*;
// A robust measurement with multiple iterations and statistical cleanup.
let result = bench_function
(
"summation_1000",
||
{
( 0..1000 ).fold( 0, | acc, x | acc + x )
}
);
println!( "Avg time: {:.2?}", result.mean_time() );
println!( "Throughput: {:.0} ops/sec", result.operations_per_second() );
// Track memory usage patterns alongside timing.
let memory_benchmark = MemoryBenchmark::new( "allocation_test" );
let ( timing, memory_stats ) = memory_benchmark.run_with_tracking
(
10,
||
{
let data = vec![ 0u8; 1024 ];
memory_benchmark.tracker.record_allocation( 1024 );
std::hint::black_box( data );
}
);
println!( "Peak memory usage: {} bytes", memory_stats.peak_usage );
Analyze: Find Insights and Regressions
Turn raw numbers into actionable insights.
use benchkit::prelude::*;
// Compare multiple implementations to find the best one.
let report = ComparativeAnalysis::new( "Hashing" )
.algorithm( "fnv", || { /* ... */ } )
.algorithm( "siphash", || { /* ... */ } )
.run();
if let Some( ( fastest_name, _ ) ) = report.fastest()
{
println!( "Fastest algorithm: {}", fastest_name );
}
// Example benchmark results
let result_a = bench_function( "test_a", || { /* ... */ } );
let result_b = bench_function( "test_b", || { /* ... */ } );
// Compare two benchmark results
let comparison = result_a.compare( &result_b );
if comparison.is_improvement()
{
println!( "Performance improved!" );
}
Generate: Create Realistic Test Data
Stop writing boilerplate to create test data. benchkit
provides generators for common scenarios.
use benchkit::prelude::*;
// Generate a comma-separated list of 100 items.
let list_data = generate_list_data( DataSize::Medium );
// Generate realistic unilang command strings for parser benchmarking.
let command_generator = DataGenerator::new()
.complexity( DataComplexity::Complex );
let commands = command_generator.generate_unilang_commands( 10 );
// Create reproducible data with a specific seed.
let mut seeded_gen = SeededGenerator::new( 42 );
let random_data = seeded_gen.random_string( 1024 );
Document: Automate Your Reports
The “documentation-first” philosophy is enabled by powerful report generation and file updating tools.
use benchkit::prelude::*;
fn main() -> Result< (), Box< dyn std::error::Error > >
{
let mut suite = BenchmarkSuite::new( "api_performance" );
suite.benchmark( "get_user", || { /* ... */ } );
suite.benchmark( "create_user", || { /* ... */ } );
let results = suite.run_analysis();
// Generate a markdown report from the results.
let markdown_report = results.generate_markdown_report().generate();
// Automatically update the "## Performance" section of a file.
let updater = MarkdownUpdater::new( "readme.md", "Performance" );
updater.update_section( &markdown_report )?;
Ok( () )
}
§The benchkit
Workflow
benchkit
is designed to make performance analysis a natural part of your development cycle.
[ 1. Write Code ] -> [ 2. Add Benchmark in `tests/` ] -> [ 3. Run `cargo test` ]
^ |
| v
[ 5. Commit Code + Perf Docs ] <- [ 4. Auto-Update `readme.md` ] <- [ Analyze Console Results ]
§Installation
Add benchkit
to your [dev-dependencies]
in Cargo.toml
.
[dev-dependencies]
# For core functionality
benchkit = "0.1"
# Or enable all features for the full toolkit
benchkit = { version = "0.1", features = [ "full" ] }
§Contributing
Contributions are welcome! benchkit
aims to be a community-driven toolkit that solves real-world benchmarking problems. Please see our contribution guidelines and open tasks.
§License
This project is licensed under the MIT License.
Modules§
- analysis
- Analysis tools for benchmark results
- comparison
- Framework and algorithm comparison utilities
- data_
generation - Advanced data generation utilities for benchmarking
- diff
- Git-style diff functionality for benchmark results
- documentation
- Documentation integration and auto-update utilities
- generators
- Data generators for benchmarking
- measurement
- Core measurement and timing functionality
- memory_
tracking - Memory allocation tracking and analysis for benchmarks
- parser_
analysis - Parser-specific analysis utilities
- parser_
data_ generation - Parser-specific data generation utilities
- plotting
- Visualization and plotting utilities for benchmark results
- prelude
- Prelude module for convenient imports
- profiling
- Memory allocation and performance profiling tools
- reporting
- Report generation and markdown integration
- scaling
- Scaling analysis tools for performance testing
- statistical
- Research-grade statistical analysis for benchmark results
- suite
- Benchmark suite management
- throughput
- Throughput calculation and analysis utilities
Macros§
- bench_
block - Measure a block of code (convenience macro)