Struct criterion::BenchmarkGroup

source · [−]

pub struct BenchmarkGroup<'a, M: Measurement> { /* private fields */ }

Expand description

Structure used to group together a set of related benchmarks, along with custom configuration settings for groups of benchmarks. All benchmarks performed using a benchmark group will be grouped together in the final report.

Examples:

#[macro_use] extern crate criterion;
use self::criterion::*;
use std::time::Duration;

fn bench_simple(c: &mut Criterion) {
    let mut group = c.benchmark_group("My Group");

    // Now we can perform benchmarks with this group
    group.bench_function("Bench 1", |b| b.iter(|| 1 ));
    group.bench_function("Bench 2", |b| b.iter(|| 2 ));
    
    // It's recommended to call group.finish() explicitly at the end, but if you don't it will
    // be called automatically when the group is dropped.
    group.finish();
}

fn bench_nested(c: &mut Criterion) {
    let mut group = c.benchmark_group("My Second Group");
    // We can override the configuration on a per-group level
    group.measurement_time(Duration::from_secs(1));

    // We can also use loops to define multiple benchmarks, even over multiple dimensions.
    for x in 0..3 {
        for y in 0..3 {
            let point = (x, y);
            let parameter_string = format!("{} * {}", x, y);
            group.bench_with_input(BenchmarkId::new("Multiply", parameter_string), &point,
                |b, (p_x, p_y)| b.iter(|| p_x * p_y));
        }
    }
    
    group.finish();
}

fn bench_throughput(c: &mut Criterion) {
    let mut group = c.benchmark_group("Summation");
     
    for size in [1024, 2048, 4096].iter() {
        // Generate input of an appropriate size...
        let input = vec![1u64, *size];

        // We can use the throughput function to tell Criterion.rs how large the input is
        // so it can calculate the overall throughput of the function. If we wanted, we could
        // even change the benchmark configuration for different inputs (eg. to reduce the
        // number of samples for extremely large and slow inputs) or even different functions.
        group.throughput(Throughput::Elements(*size as u64));

        group.bench_with_input(BenchmarkId::new("sum", *size), &input,
            |b, i| b.iter(|| i.iter().sum::<u64>()));
        group.bench_with_input(BenchmarkId::new("fold", *size), &input,
            |b, i| b.iter(|| i.iter().fold(0u64, |a, b| a + b)));
    }

    group.finish();
}

criterion_group!(benches, bench_simple, bench_nested, bench_throughput);
criterion_main!(benches);

Implementations

source

impl<'a, M: Measurement> BenchmarkGroup<'a, M>

source

pub fn sample_size(&mut self, n: usize) -> &mut Self

Changes the size of the sample for this benchmark

A bigger sample should yield more accurate results if paired with a sufficiently large measurement time.

Sample size must be at least 10.

Panics

Panics if n < 10.

source

pub fn warm_up_time(&mut self, dur: Duration) -> &mut Self

Changes the warm up time for this benchmark

Panics

Panics if the input duration is zero

source

pub fn measurement_time(&mut self, dur: Duration) -> &mut Self

Changes the target measurement time for this benchmark group.

Criterion will attempt to spent approximately this amount of time measuring each benchmark on a best-effort basis. If it is not possible to perform the measurement in the requested time (eg. because each iteration of the benchmark is long) then Criterion will spend as long as is needed to collect the desired number of samples. With a longer time, the measurement will become more resilient to interference from other programs.

Panics

Panics if the input duration is zero

source

pub fn nresamples(&mut self, n: usize) -> &mut Self

Changes the number of resamples for this benchmark group

Number of resamples to use for the bootstrap

A larger number of resamples reduces the random sampling errors which are inherent to the bootstrap method, but also increases the analysis time.

Panics

Panics if the number of resamples is set to zero

source

pub fn noise_threshold(&mut self, threshold: f64) -> &mut Self

Changes the noise threshold for benchmarks in this group. The noise threshold is used to filter out small changes in performance from one run to the next, even if they are statistically significant. Sometimes benchmarking the same code twice will result in small but statistically significant differences solely because of noise. This provides a way to filter out some of these false positives at the cost of making it harder to detect small changes to the true performance of the benchmark.

The default is 0.01, meaning that changes smaller than 1% will be ignored.

Panics

Panics if the threshold is set to a negative value

source

pub fn confidence_level(&mut self, cl: f64) -> &mut Self

Changes the confidence level for benchmarks in this group. The confidence level is the desired probability that the true runtime lies within the estimated confidence interval. The default is 0.95, meaning that the confidence interval should capture the true value 95% of the time.

Panics

Panics if the confidence level is set to a value outside the (0, 1) range

source

pub fn significance_level(&mut self, sl: f64) -> &mut Self

Changes the significance level for benchmarks in this group. This is used to perform a hypothesis test to see if the measurements from this run are different from the measured performance of the last run. The significance level is the desired probability that two measurements of identical code will be considered ‘different’ due to noise in the measurements. The default value is 0.05, meaning that approximately 5% of identical benchmarks will register as different due to noise.

This presents a trade-off. By setting the significance level closer to 0.0, you can increase the statistical robustness against noise, but it also weakens Criterion.rs’ ability to detect small but real changes in the performance. By setting the significance level closer to 1.0, Criterion.rs will be more able to detect small true changes, but will also report more spurious differences.