Crate benchkit

Source
Expand description

Lightweight benchmarking toolkit focused on practical performance analysis and report generation.

§benchkit

docs.rs discord

Practical, Documentation-First Benchmarking for Rust.

benchkit is a lightweight toolkit for performance analysis, born from the hard-learned lessons of optimizing high-performance libraries. It rejects rigid, all-or-nothing frameworks in favor of flexible, composable tools that integrate seamlessly into your existing workflow.

§The Benchmarking Dilemma

In Rust, developers often face a frustrating choice:

  1. The Heavy Framework (criterion): Statistically powerful, but forces a rigid structure (benches/), complex setup, and produces reports that are difficult to integrate into your project’s documentation. You must adapt your project to the framework.
  2. The Manual Approach (std::time): Simple to start, but statistically naive. It leads to boilerplate, inconsistent measurements, and conclusions that are easily skewed by system noise.

benchkit offers a third way.

§A Toolkit, Not a Framework

This is the core philosophy of benchkit. It doesn’t impose a workflow; it provides a set of professional, composable tools that you can use however you see fit.

  • âś… Integrate Anywhere: Write benchmarks in your test files, examples, or binaries. No required directory structure.
  • âś… Documentation-First: Treat performance reports as a first-class part of your documentation, with tools to automatically keep them in sync with your code.
  • âś… Practical Focus: Surface the key metrics needed for optimization decisions, hiding deep statistical complexity until you ask for it.
  • âś… Zero Setup: Start measuring performance in minutes with a simple, intuitive API.

§🚀 Quick Start: Compare, Analyze, and Document

This example demonstrates the core benchkit workflow: comparing two algorithms and automatically updating a performance section in your readme.md.

1. Add to dev-dependencies in Cargo.toml:

[dev-dependencies]
benchkit = { version = "0.1", features = [ "full" ] }

2. Create a benchmark in your tests directory:

// In tests/performance_test.rs
#![ cfg( feature = "integration" ) ]
use benchkit::prelude::*;

fn generate_data( size : usize ) -> Vec< u32 >
{
  ( 0..size ).map( | x | x as u32 ).collect()
}

#[ test ]
fn update_readme_performance_docs()
{
  let mut comparison = ComparativeAnalysis::new( "Sorting Algorithms" );
  let data = generate_data( 1000 );

  // Benchmark the first algorithm
  comparison = comparison.algorithm
  (
    "std_stable_sort",
    {
      let mut d = data.clone();
      move ||
      {
        d.sort();
      }
    }
  );

  // Benchmark the second algorithm
  comparison = comparison.algorithm
  (
    "std_unstable_sort",
    {
      let mut d = data.clone();
      move ||
      {
        d.sort_unstable();
      }
    }
  );

  // Run the comparison and update the documentation
  let report = comparison.run();
  let markdown = report.to_markdown();

  let updater = MarkdownUpdater::new( "readme.md", "Performance" );
  updater.update_section( &markdown ).unwrap();
}

3. Add a placeholder section to your readme.md:

## Performance

## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |

### Key Insights

- **Fastest operation**: create_user (36.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |

### Key Insights

- **Fastest operation**: get_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |
| create_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |

### Key Insights

- **Fastest operation**: get_user (64.00ns)
- **Performance range**: 1.0x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |

### Key Insights

- **Fastest operation**: create_user (40.00ns)
- **Performance range**: 1.0x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 24.00ns | 41666667 | 0.00ns | 40.00ns | 21.00ns |
| create_user | 28.00ns | 35714286 | 0.00ns | 40.00ns | 19.00ns |

### Key Insights

- **Fastest operation**: get_user (24.00ns)
- **Performance range**: 1.2x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| get_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |

### Key Insights

- **Fastest operation**: create_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| get_user | 88.00ns | 11363636 | 80.00ns | 120.00ns | 17.00ns |

### Key Insights

- **Fastest operation**: create_user (84.00ns)
- **Performance range**: 1.0x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| create_user | 92.00ns | 10869565 | 80.00ns | 120.00ns | 19.00ns |

### Key Insights

- **Fastest operation**: get_user (84.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## Performance

## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |

### Key Insights

- **Fastest operation**: create_user (36.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| create_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |

### Key Insights

- **Fastest operation**: get_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |
| create_user | 64.00ns | 15625000 | 40.00ns | 80.00ns | 21.00ns |

### Key Insights

- **Fastest operation**: get_user (64.00ns)
- **Performance range**: 1.0x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |
| get_user | 40.00ns | 25000000 | 40.00ns | 40.00ns | 0.00ns |

### Key Insights

- **Fastest operation**: create_user (40.00ns)
- **Performance range**: 1.0x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 24.00ns | 41666667 | 0.00ns | 40.00ns | 21.00ns |
| create_user | 28.00ns | 35714286 | 0.00ns | 40.00ns | 19.00ns |

### Key Insights

- **Fastest operation**: get_user (24.00ns)
- **Performance range**: 1.2x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 32.00ns | 31250000 | 0.00ns | 40.00ns | 17.00ns |
| get_user | 36.00ns | 27777778 | 0.00ns | 40.00ns | 13.00ns |

### Key Insights

- **Fastest operation**: create_user (32.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| create_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| get_user | 88.00ns | 11363636 | 80.00ns | 120.00ns | 17.00ns |

### Key Insights

- **Fastest operation**: create_user (84.00ns)
- **Performance range**: 1.0x difference between fastest and slowest



## api_performance Results

| Benchmark | Mean Time | Ops/sec | Min | Max | Std Dev |
|-----------|-----------|---------|-----|-----|----------|
| get_user | 84.00ns | 11904762 | 80.00ns | 120.00ns | 13.00ns |
| create_user | 92.00ns | 10869565 | 80.00ns | 120.00ns | 19.00ns |

### Key Insights

- **Fastest operation**: get_user (84.00ns)
- **Performance range**: 1.1x difference between fastest and slowest



## đź§° What's in the Toolkit?

`benchkit` provides a suite of composable tools. Use only what you need.

<details>
<summary><strong>Measure: Core Timing and Profiling</strong></summary>

At its heart, `benchkit` provides simple and accurate measurement primitives.

```rust
use benchkit::prelude::*;

// A robust measurement with multiple iterations and statistical cleanup.
let result = bench_function
(
  "summation_1000",
  ||
  {
    ( 0..1000 ).fold( 0, | acc, x | acc + x )
  }
);
println!( "Avg time: {:.2?}", result.mean_time() );
println!( "Throughput: {:.0} ops/sec", result.operations_per_second() );

// Track memory usage patterns alongside timing.
let memory_benchmark = MemoryBenchmark::new( "allocation_test" );
let ( timing, memory_stats ) = memory_benchmark.run_with_tracking
(
  10,
  ||
  {
    let data = vec![ 0u8; 1024 ];
    memory_benchmark.tracker.record_allocation( 1024 );
    std::hint::black_box( data );
  }
);
println!( "Peak memory usage: {} bytes", memory_stats.peak_usage );
Analyze: Find Insights and Regressions

Turn raw numbers into actionable insights.

use benchkit::prelude::*;

// Compare multiple implementations to find the best one.
let report = ComparativeAnalysis::new( "Hashing" )
.algorithm( "fnv", || { /* ... */ } )
.algorithm( "siphash", || { /* ... */ } )
.run();

if let Some( ( fastest_name, _ ) ) = report.fastest()
{
  println!( "Fastest algorithm: {}", fastest_name );
}

// Example benchmark results
let result_a = bench_function( "test_a", || { /* ... */ } );
let result_b = bench_function( "test_b", || { /* ... */ } );

// Compare two benchmark results
let comparison = result_a.compare( &result_b );
if comparison.is_improvement()
{
  println!( "Performance improved!" );
}
Generate: Create Realistic Test Data

Stop writing boilerplate to create test data. benchkit provides generators for common scenarios.

use benchkit::prelude::*;

// Generate a comma-separated list of 100 items.
let list_data = generate_list_data( DataSize::Medium );

// Generate realistic unilang command strings for parser benchmarking.
let command_generator = DataGenerator::new()
.complexity( DataComplexity::Complex );
let commands = command_generator.generate_unilang_commands( 10 );

// Create reproducible data with a specific seed.
let mut seeded_gen = SeededGenerator::new( 42 );
let random_data = seeded_gen.random_string( 1024 );
Document: Automate Your Reports

The “documentation-first” philosophy is enabled by powerful report generation and file updating tools.

use benchkit::prelude::*;

fn main() -> Result< (), Box< dyn std::error::Error > >
{
  let mut suite = BenchmarkSuite::new( "api_performance" );
  suite.benchmark( "get_user", || { /* ... */ } );
  suite.benchmark( "create_user", || { /* ... */ } );
  let results = suite.run_analysis();

  // Generate a markdown report from the results.
  let markdown_report = results.generate_markdown_report().generate();

  // Automatically update the "## Performance" section of a file.
  let updater = MarkdownUpdater::new( "readme.md", "Performance" );
  updater.update_section( &markdown_report )?;
  
  Ok( () )
}

§The benchkit Workflow

benchkit is designed to make performance analysis a natural part of your development cycle.

[ 1. Write Code ] -> [ 2. Add Benchmark in `tests/` ] -> [ 3. Run `cargo test` ]
       ^                                                              |
       |                                                              v
[ 5. Commit Code + Perf Docs ] <- [ 4. Auto-Update `readme.md` ] <- [ Analyze Console Results ]

§Installation

Add benchkit to your [dev-dependencies] in Cargo.toml.

[dev-dependencies]
# For core functionality
benchkit = "0.1"

# Or enable all features for the full toolkit
benchkit = { version = "0.1", features = [ "full" ] }

§Contributing

Contributions are welcome! benchkit aims to be a community-driven toolkit that solves real-world benchmarking problems. Please see our contribution guidelines and open tasks.

§License

This project is licensed under the MIT License.

Modules§

analysis
Analysis tools for benchmark results
comparison
Framework and algorithm comparison utilities
data_generation
Advanced data generation utilities for benchmarking
diff
Git-style diff functionality for benchmark results
documentation
Documentation integration and auto-update utilities
generators
Data generators for benchmarking
measurement
Core measurement and timing functionality
memory_tracking
Memory allocation tracking and analysis for benchmarks
parser_analysis
Parser-specific analysis utilities
parser_data_generation
Parser-specific data generation utilities
plotting
Visualization and plotting utilities for benchmark results
prelude
Prelude module for convenient imports
profiling
Memory allocation and performance profiling tools
reporting
Report generation and markdown integration
scaling
Scaling analysis tools for performance testing
statistical
Research-grade statistical analysis for benchmark results
suite
Benchmark suite management
throughput
Throughput calculation and analysis utilities

Macros§

bench_block
Measure a block of code (convenience macro)