criterion-table 0.1.1

Generate markdown comparison tables from cargo-criterion benchmark output
Documentation

criterion_compare

Crate Docs

Generate markdown comparison tables from Cargo Criterion benchmark output.

Currently, the tool is limited to Github Flavored Markdown (GFM), but adding new output types is simple.

Installation

# If you don't have it already
cargo install cargo-criterion

# This project
cargo install criterion-table

Usage

  1. Ensure your benchmarks meet these basic criteria:
  • Benchmark IDs are formatted in two to three sections separated by forward slashes (/)
    • The sections are used like this: <table_name>/<column_name>/[row_name]
    • Case is not currently altered, so set appropriately for display
    • Row name is the only optional field, and if left blank, all results will be a single blank row
    • If using a very basic benchmark_function you would only get a column name, which isn't valid
    • If using benchmark groups you will get two sections automatically
    • If using benchmark groups and BenchmarkId you will get all three sections automatically
  • Benchmark data is not reordered, so ensure they execute in the order desired
    • Tables are ordered based on the order they are seen in the data (execution order)
    • The first column seen in each row will be the baseline everything else in that row is compared to, so benchmark execution order matters

Benchmark Example 1 - Manual ID

use criterion::{black_box, criterion_group, criterion_main, Criterion};

#[inline]
fn fibonacci(n: u64) -> u64 {
  match n {
    0 => 1,
    1 => 1,
    n => fibonacci(n-1) + fibonacci(n-2),
  }
}

pub fn criterion_benchmark(c: &mut Criterion) {
    let id = "Fibonacci/Recursive Fib/20";
    c.bench_function(id, |b| b.iter(|| fibonacci(black_box(20))));
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);

Benchmark Example 2 - Benchmark Group with Parameters

use criterion::{black_box, BenchmarkId, criterion_group, criterion_main, 
                Criterion};

#[inline]
fn fibonacci(n: u64) -> u64 {
  match n {
    0 => 1,
    1 => 1,
    n => fibonacci(n-1) + fibonacci(n-2),
  }
}

pub fn criterion_benchmark(c: &mut Criterion) {
    let mut group = c.benchmark_group("Fibonacci");
    
    for row in vec![10, 20] {
        let id = BenchmarkId::new("Recursive Fib", row);
        group.bench_with_input(id, &row, |b, row| {
            b.iter(|| fibonacci(black_box(*row)))
        });
    }
    
    group.finish();
}

criterion_group!(benches, criterion_benchmark);
criterion_main!(benches);
  1. Run Benchmarks and Generate Markdown

This can be done in a couple of different ways:

Single Step

This method ensures all benchmarks are included in one step

# Run all benchmarks and convert into the markdown all in one step
cargo criterion --message-format=json | criterion-table > BENCHMARKS.md

Multiple Steps

This method allows better control of order and which benchmarks are included

# Execute only the desired benchmarks
cargo criterion --bench recursive_fib --message-format=json > recursive_fib.json
cargo criterion --bench iterative_fib --message-format=json > iterative_fib.json

# Reorder before converting into markdown
cat iterative_fib.json recursive_fib.json | criterion-table > BENCHMARKS.md

Adding New Output File Types

Currently, the tool is hardcoded to GFM, but it is easy to add a new output type via the Formatter trait by creating your own new binary project

  1. Add this crate to your binary project as a library
[dependencies]
criterion-table = "0.1"
  1. Create a new type and implement criterion_table::Formatter
use criterion_table::{ColumnInfo, Comparison, TimeUnit};
use flexstr::FlexStr;

pub trait Formatter {
    fn start(&mut self, buffer: &mut String, tables: &[&FlexStr]);

    fn end(&mut self, buffer: &mut String);

    fn start_table(&mut self, buffer: &mut String, name: &FlexStr, columns: &[ColumnInfo]);

    fn end_table(&mut self, buffer: &mut String);

    fn start_row(&mut self, buffer: &mut String, name: &FlexStr, max_width: usize);

    fn end_row(&mut self, buffer: &mut String);

    fn used_column(&mut self, buffer: &mut String, time: TimeUnit, pct: Comparison, max_width: usize);

    fn unused_column(&mut self, buffer: &mut String, max_width: usize);
}
  1. Create a process function that takes cargo-criterion raw JSON as input and outputs your format as a String:

NOTE: Replace GFMFormatter with your new formatter below

use std::error::Error;
use std::io::Read;

// This would be replaced with your formatter
use criterion_table::formatter::GFMFormatter;
use criterion_table::{CriterionTableData, RawCriterionData};

fn process(r: impl Read) -> Result<String, Box<dyn Error>> {
    let raw_data = RawCriterionData::from_reader(r)?;
    let data = CriterionTableData::from_raw(&raw_data)?;
    
    // Pass your formatter to `make_tables` below
    Ok(data.make_tables(GFMFormatter))
}
  1. Save the returned String to the file type of your formatter

License

This project is licensed optionally under either: