criterion-markdown

criterion-markdown reads Criterion benchmark JSON output from target/criterion and renders a markdown summary table.

What It Does

• Walks a Criterion result directory and discovers benchmark runs.
• Reads benchmark.json and estimates.json from the latest run and selected baseline.
• Produces grouped markdown tables with human-readable timings and change indicators.
• Computes improvements and regressions at render time, so any saved Criterion baseline can be selected.

Usage

Use Renderer to select datasets, filter benchmarks, and configure the output:

use criterion_markdown::{ChangeThresholds, Renderer};

fn main() -> anyhow::Result<()> {
        let thresholds = ChangeThresholds::default()
                .improvement_ratio(1.1)
                .strong_improvement_ratio(1.5)
                .regression_ratio(0.95);
        let markdown = Renderer::new("target/criterion")
                .candidate("new")
                .baseline_root("artifacts/criterion")
                .baseline("main")
                .benchmark("group/benchmark/1")
                .benchmarks(["group/benchmark/2", "group/benchmark/3"])
                .change_thresholds(thresholds)
                .summary_limit(5)
                .title("Benchmark Results")
                .collapsible(true)
                .render()?;

        println!("{markdown}");
        Ok(())
}

The builder defaults to the new candidate, all benchmark entries, the title Benchmarks, and non-collapsible output. When no baseline is specified, it uses Criterion's default base dataset if present and otherwise omits comparison information. An explicitly selected baseline that cannot be found returns an error. Calls to benchmark and benchmarks are additive. Use baseline to select a different dataset, title to configure the top-level Markdown heading or <summary> label, and collapsible(true) to wrap the report in a <details> element.

The summary appears before the detailed benchmark tables. It lists the top three improvements and top three regressions by default, using the same configured thresholds as the table indicators and omitting either category when it has no entries. If no compared benchmark crosses either threshold, the summary says so explicitly. Use summary_limit to set the maximum number shown in each category; a limit of 0 omits the summary.

Change thresholds use the ratio baseline time / candidate time. The defaults classify changes as follows:

• ratio <= 0.9: ❌ regression
• 0.9 < ratio < 1.1: ➖ neutral
• 1.1 <= ratio < 1.8: ↗️ improvement
• ratio >= 1.8: 🚀 strong improvement

Invalid or incorrectly ordered threshold configurations return an error from render.

baseline_root can point to a separate Criterion output tree, such as a downloaded CI artifact. It is also where the renderer looks for the default base dataset when no baseline is specified. For each candidate benchmark, the renderer reads the baseline from the same relative benchmark path beneath that root.

The existing free functions remain available as convenience entrypoints:

• criterion_markdown::render(criterion_dir, allowlist) renders the new candidate and compares it with base when available.
• criterion_markdown::render_with_options(criterion_dir, allowlist, &options) additionally configures an optional baseline, title, and collapsible output through RenderOptions.

The change point estimate is computed with the same formula Criterion uses: candidate_mean / baseline_mean - 1. Selecting the same candidate and baseline Criterion used for a run therefore produces the same point estimate as its change/estimates.json output. If the selected baseline is absent for every benchmark, the report omits comparison information. If the baseline exists for only some benchmarks, entries without it render --- for their change.

For comparisons rendered after the benchmark run, prefer a stable named baseline created with Criterion's --save-baseline <name> option. Criterion's default save mode can replace base with the new measurements after computing its change, so that directory may no longer contain the historical data used by the precomputed comparison.

How This Differs From criterion-table

• This crate reads benchmark data directly from the JSON files generated by Criterion in target/criterion.
• It does not depend on cargo-criterion.
• This crate is a library you can embed and call from your own project code; criterion-table is primarily used as a standalone binary tool.
• You can point it at an existing Criterion output directory and render markdown without changing how benchmarks are run.

Development

Run checks locally:

cargo +nightly fmt --check
cargo test
cargo clippy --all-targets -- -D warnings