benchmark-rs 0.1.3

//! A benchmarking library for Rust package authors.
//!
//! Benchmark-rs crate provides tools for Rust package authors to evaluate performance of their implementation for
//! varying workloads with varying configurations, find performance regressions between versions and also some crude
//! facilities to measure execution times of code blocks.
//!
//! For example, if you implement a concurrent algorithm with an expectation of reduced execution time when more
//! CPU cores are added, benchmark-rs will help validate that the concurrency has the desired effect and that there
//! is no anomaly for varying data sets, that is, the algorithm performs consistently within boundaries defined for it.
//!
//! However, this doesn't have to be a fancy algorithm that we want to test, some banal looking code can exhibit
//! radically different behavior for different data sets, depending on database queries, buffer sizes, number of
//! open files, etc. We may want to compare the behavior of our code with different querying strategies or different
//! buffer sizes for a range of data sets to find the best fit for our use-case.
//!
//!
//! Another common use-case is to verify that there is no regression in performance introduced by the new code. Benchmark-rs
//! supports comparison to previous results with user defined equality threshold.
//!
//! # Design
//!
//! Benchmark-rs is designed to evaluate performance of subjects processing a set of workloads with modifiable configurations.
//! For example if we write a protocol parser we may define our workload points as 10MB, 20MB, ..., 100MB input sizes. Workloads
//! are user defined in benchmark-rs. Any type `W: Clone + Display` can be used to specify workloads. It could
//! be an integer that specifies the size of the workload, a path to a file or a key that we can use
//! to fetch the workload from the benchmark configuration. The `Display` is required
//! for the workload point to produce the key in result series, so it is recommended to keep it short and descriptive.
//! See examples below.
//!
//! Each benchmark is repeated `repeat` times for every workload point after it was ramped up for that point. Upon
//! completion the summary is available as JSON or as CSV.
//!
//! # Issues
//! Issues are welcome and appreciated. Please submit to https://github.com/navigatorsguild/benchmark-rs/issues
//!
//! # Examples
//! A real world example can be found at
//! [command-executor](https://github.com/navigatorsguild/command-executor) project
//! [blocking_queue.rs](https://github.com/navigatorsguild/command-executor/blob/main/benches/blocking_queue.rs)
//! benchmark and at [Benchmarks](https://github.com/navigatorsguild/command-executor/wiki/Benchmarks)
//! wiki page which was built from the generated data.
//!
//! ![link](https://user-images.githubusercontent.com/122003456/235414598-727d804a-b8ad-4520-871b-5fd8be33bf44.png)
//!
//! ## Simple Benchmark
//! A simple benchmark that measures execution time for increasing workloads. In this case the workload is simulated by
//! by a `u64` value passed to `thread::sleep` function
//! ```
//! use std::thread;
//! use std::time::Duration;
//! use benchmark_rs::benchmarks::Benchmarks;
//! use benchmark_rs::stopwatch::StopWatch;
//!
//! fn example(_stop_watch: &mut StopWatch, _config: &str, work: u64) -> Result<(), anyhow::Error> {
//!     thread::sleep(Duration::from_millis(work));
//!     Ok(())
//! }
//!
//! fn main() -> Result<(), anyhow::Error> {
//!     let mut benchmarks = Benchmarks::new("Example");
//!     benchmarks.add("A Simple Benchmark", example, "No Configuration", (1..=10).collect(), 2, 1)?;
//!     benchmarks.run()?;
//!
//!     let summary = benchmarks.summary_as_json();
//!     println!("Summary: {summary}");
//!     Ok(())
//! }
//! ```
//!
//! ## Benchmark Workloads
//!
//! A more complex example that shows how to use Benchmark configuration and how to control the
//! stopwatch from within the benchmark to avoid measuring the housekeeping tasks.
//!
//! <details>
//!  <summary>Benchmark Workloads Example</summary>
//!
//! ```
//! use std::collections::BTreeMap;
//! use std::fmt::{Display, Formatter};
//! use std::thread;
//! use std::time::Duration;
//!
//! use benchmark_rs::benchmarks::Benchmarks;
//! use benchmark_rs::stopwatch::StopWatch;
//!
//! #[derive(Clone)]
//! struct Config {
//!     // simulate available resources - CPU cores, memory buffers, etc.
//!     pub resources: u32,
//!     pub workloads: BTreeMap<u64, Duration>,
//! }
//!
//! impl Display for Config {
//!     fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
//!         let keys: Vec<String> = self.workloads.keys().map(|k| k.to_string()).collect();
//!         write!(f, "{}", keys.join(", "))
//!     }
//! }
//!
//! fn example(stop_watch: &mut StopWatch, config: Config, work: u64) -> Result<(), anyhow::Error> {
//!     stop_watch.pause();
//!     // perform potentially lengthy preparation that will not reflect in the measurement
//!     // ...
//!     // fetch the workload definition from configuration associated with the 'work' point
//!     let sleep_time = config.workloads.get(&work).unwrap().clone();
//!     // resume the stopwatch to measure the actual work
//!     stop_watch.resume();
//!     // perform measured computation
//!     thread::sleep(sleep_time / config.resources);
//!     stop_watch.pause();
//!     // perform potentially lengthy cleanup
//!     // ...
//!     Ok(())
//! }
//!
//! fn main() -> Result<(), anyhow::Error> {
//!     let mut benchmarks = Benchmarks::new("benchmark-workloads");
//!     let workloads: BTreeMap<u64, Duration> =
//!         (0..=10).map(|i| (i, Duration::from_millis(i))).collect();
//!     benchmarks.add(
//!         "benchmark-workload-1",
//!         example,
//!         Config {
//!             resources: 1,
//!             workloads: workloads.clone(),
//!         },
//!         (1..=10).collect(),
//!         2,
//!         1,
//!     )?;
//!
//!     benchmarks.add(
//!         "benchmark-workload-2",
//!         example,
//!         Config {
//!             resources: 2,
//!             workloads: workloads.clone(),
//!         },
//!         (1..=10).collect(),
//!         2,
//!         1,
//!     )?;
//!
//!     benchmarks.run()?;
//!
//!     let summary = benchmarks.summary_as_json();
//!     println!("Benchmark summary in JSON format.");
//!     println!("Summary:");
//!     println!("{summary}");
//!     println!();
//!
//!     println!("Benchmark series in CSV format.");
//!     let csv_data = benchmarks.summary_as_csv(true, false);
//!     for (k, v) in csv_data {
//!         println!("Benchmark name: {k}");
//!         for line in v {
//!             println!("{line}")
//!         }
//!         println!();
//!     }
//!
//!     Ok(())
//! }
//! ```
//! </details>
//!
//! <details>
//!  <summary>Benchmark Workloads Summary</summary>
//!
//! ```json
//! {
//!   "name": "benchmark-workloads",
//!   "created_at": "2023-08-13 04:09:13.923036",
//!   "series": {
//!     "benchmark-workload-2": {
//!       "name": "benchmark-workload-2",
//!       "config": "0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10",
//!       "runs": [
//!         [
//!           "1",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 639791,
//!             "min_sec": 0.000639791,
//!             "min_str": "00:00:00.000",
//!             "max_nanos": 644250,
//!             "max_sec": 0.00064425,
//!             "max_str": "00:00:00.000",
//!             "median_nanos": 642020,
//!             "median_sec": 0.00064202,
//!             "median_str": "00:00:00.000",
//!             "std_dev": 3152.9891373108153,
//!             "std_dev_sec": 3.1529891373108154e-6,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "2",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 1264124,
//!             "min_sec": 0.001264124,
//!             "min_str": "00:00:00.001",
//!             "max_nanos": 1264250,
//!             "max_sec": 0.00126425,
//!             "max_str": "00:00:00.001",
//!             "median_nanos": 1264187,
//!             "median_sec": 0.001264187,
//!             "median_str": "00:00:00.001",
//!             "std_dev": 89.09545442950498,
//!             "std_dev_sec": 8.909545442950498e-8,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "3",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 1891833,
//!             "min_sec": 0.001891833,
//!             "min_str": "00:00:00.001",
//!             "max_nanos": 1892875,
//!             "max_sec": 0.001892875,
//!             "max_str": "00:00:00.001",
//!             "median_nanos": 1892354,
//!             "median_sec": 0.001892354,
//!             "median_str": "00:00:00.001",
//!             "std_dev": 736.8052659963826,
//!             "std_dev_sec": 7.368052659963826e-7,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "4",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 2566417,
//!             "min_sec": 0.002566417,
//!             "min_str": "00:00:00.002",
//!             "max_nanos": 2576416,
//!             "max_sec": 0.002576416,
//!             "max_str": "00:00:00.002",
//!             "median_nanos": 2571416,
//!             "median_sec": 0.002571416,
//!             "median_str": "00:00:00.002",
//!             "std_dev": 7070.360705084288,
//!             "std_dev_sec": 7.0703607050842884e-6,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "5",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 2928666,
//!             "min_sec": 0.002928666,
//!             "min_str": "00:00:00.002",
//!             "max_nanos": 3174917,
//!             "max_sec": 0.003174917,
//!             "max_str": "00:00:00.003",
//!             "median_nanos": 3051791,
//!             "median_sec": 0.003051791,
//!             "median_str": "00:00:00.003",
//!             "std_dev": 174125.75197396852,
//!             "std_dev_sec": 0.00017412575197396852,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "6",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 3377791,
//!             "min_sec": 0.003377791,
//!             "min_str": "00:00:00.003",
//!             "max_nanos": 3784625,
//!             "max_sec": 0.003784625,
//!             "max_str": "00:00:00.003",
//!             "median_nanos": 3581208,
//!             "median_sec": 0.003581208,
//!             "median_str": "00:00:00.003",
//!             "std_dev": 287675.0802172479,
//!             "std_dev_sec": 0.00028767508021724787,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "7",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 4404499,
//!             "min_sec": 0.004404499,
//!             "min_str": "00:00:00.004",
//!             "max_nanos": 4915541,
//!             "max_sec": 0.004915541,
//!             "max_str": "00:00:00.004",
//!             "median_nanos": 4660020,
//!             "median_sec": 0.00466002,
//!             "median_str": "00:00:00.004",
//!             "std_dev": 361361.26367113565,
//!             "std_dev_sec": 0.00036136126367113564,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "8",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 4882584,
//!             "min_sec": 0.004882584,
//!             "min_str": "00:00:00.004",
//!             "max_nanos": 5029209,
//!             "max_sec": 0.005029209,
//!             "max_str": "00:00:00.005",
//!             "median_nanos": 4955896,
//!             "median_sec": 0.004955896,
//!             "median_str": "00:00:00.004",
//!             "std_dev": 103679.53179147754,
//!             "std_dev_sec": 0.00010367953179147753,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "9",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 5638959,
//!             "min_sec": 0.005638959,
//!             "min_str": "00:00:00.005",
//!             "max_nanos": 5643416,
//!             "max_sec": 0.005643416,
//!             "max_str": "00:00:00.005",
//!             "median_nanos": 5641187,
//!             "median_sec": 0.005641187,
//!             "median_str": "00:00:00.005",
//!             "std_dev": 3151.5749237484424,
//!             "std_dev_sec": 3.1515749237484423e-6,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "10",
//!           {
//!             "name": "benchmark-workload-2",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 5086250,
//!             "min_sec": 0.00508625,
//!             "min_str": "00:00:00.005",
//!             "max_nanos": 5587292,
//!             "max_sec": 0.005587292,
//!             "max_str": "00:00:00.005",
//!             "median_nanos": 5336771,
//!             "median_sec": 0.005336771,
//!             "median_str": "00:00:00.005",
//!             "std_dev": 354290.19585927017,
//!             "std_dev_sec": 0.00035429019585927015,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ]
//!       ]
//!     },
//!     "benchmark-workload-1": {
//!       "name": "benchmark-workload-1",
//!       "config": "0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10",
//!       "runs": [
//!         [
//!           "1",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 1259375,
//!             "min_sec": 0.001259375,
//!             "min_str": "00:00:00.001",
//!             "max_nanos": 1276125,
//!             "max_sec": 0.001276125,
//!             "max_str": "00:00:00.001",
//!             "median_nanos": 1267750,
//!             "median_sec": 0.00126775,
//!             "median_str": "00:00:00.001",
//!             "std_dev": 11844.038584874672,
//!             "std_dev_sec": 0.000011844038584874672,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "2",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 2513584,
//!             "min_sec": 0.002513584,
//!             "min_str": "00:00:00.002",
//!             "max_nanos": 2520875,
//!             "max_sec": 0.002520875,
//!             "max_str": "00:00:00.002",
//!             "median_nanos": 2517229,
//!             "median_sec": 0.002517229,
//!             "median_str": "00:00:00.002",
//!             "std_dev": 5155.515541631118,
//!             "std_dev_sec": 5.155515541631118e-6,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "3",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 3755750,
//!             "min_sec": 0.00375575,
//!             "min_str": "00:00:00.003",
//!             "max_nanos": 3784000,
//!             "max_sec": 0.003784,
//!             "max_str": "00:00:00.003",
//!             "median_nanos": 3769875,
//!             "median_sec": 0.003769875,
//!             "median_str": "00:00:00.003",
//!             "std_dev": 19975.766568519968,
//!             "std_dev_sec": 0.00001997576656851997,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "4",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 5003833,
//!             "min_sec": 0.005003833,
//!             "min_str": "00:00:00.005",
//!             "max_nanos": 5030084,
//!             "max_sec": 0.005030084,
//!             "max_str": "00:00:00.005",
//!             "median_nanos": 5016958,
//!             "median_sec": 0.005016958,
//!             "median_str": "00:00:00.005",
//!             "std_dev": 18562.26011292806,
//!             "std_dev_sec": 0.00001856226011292806,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "5",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 5518500,
//!             "min_sec": 0.0055185,
//!             "min_str": "00:00:00.005",
//!             "max_nanos": 6283084,
//!             "max_sec": 0.006283084,
//!             "max_str": "00:00:00.006",
//!             "median_nanos": 5900792,
//!             "median_sec": 0.005900792,
//!             "median_str": "00:00:00.005",
//!             "std_dev": 540642.5311867353,
//!             "std_dev_sec": 0.0005406425311867353,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "6",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 6942292,
//!             "min_sec": 0.006942292,
//!             "min_str": "00:00:00.006",
//!             "max_nanos": 7541458,
//!             "max_sec": 0.007541458,
//!             "max_str": "00:00:00.007",
//!             "median_nanos": 7241875,
//!             "median_sec": 0.007241875,
//!             "median_str": "00:00:00.007",
//!             "std_dev": 423674.3416564189,
//!             "std_dev_sec": 0.00042367434165641895,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "7",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 8784417,
//!             "min_sec": 0.008784417,
//!             "min_str": "00:00:00.008",
//!             "max_nanos": 8812875,
//!             "max_sec": 0.008812875,
//!             "max_str": "00:00:00.008",
//!             "median_nanos": 8798646,
//!             "median_sec": 0.008798646,
//!             "median_str": "00:00:00.008",
//!             "std_dev": 20122.84477900677,
//!             "std_dev_sec": 0.00002012284477900677,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "8",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 9426875,
//!             "min_sec": 0.009426875,
//!             "min_str": "00:00:00.009",
//!             "max_nanos": 11596125,
//!             "max_sec": 0.011596125,
//!             "max_str": "00:00:00.011",
//!             "median_nanos": 10511500,
//!             "median_sec": 0.0105115,
//!             "median_str": "00:00:00.010",
//!             "std_dev": 1533891.3850889183,
//!             "std_dev_sec": 0.0015338913850889183,
//!             "std_dev_str": "00:00:00.001"
//!           }
//!         ],
//!         [
//!           "9",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 9498333,
//!             "min_sec": 0.009498333,
//!             "min_str": "00:00:00.009",
//!             "max_nanos": 9953333,
//!             "max_sec": 0.009953333,
//!             "max_str": "00:00:00.009",
//!             "median_nanos": 9725833,
//!             "median_sec": 0.009725833,
//!             "median_str": "00:00:00.009",
//!             "std_dev": 321733.5854398791,
//!             "std_dev_sec": 0.0003217335854398791,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ],
//!         [
//!           "10",
//!           {
//!             "name": "benchmark-workload-1",
//!             "ramp_up": 1,
//!             "repeat": 2,
//!             "min_nanos": 11674000,
//!             "min_sec": 0.011674,
//!             "min_str": "00:00:00.011",
//!             "max_nanos": 12627167,
//!             "max_sec": 0.012627167,
//!             "max_str": "00:00:00.012",
//!             "median_nanos": 12150583,
//!             "median_sec": 0.012150583,
//!             "median_str": "00:00:00.012",
//!             "std_dev": 673990.849303238,
//!             "std_dev_sec": 0.0006739908493032379,
//!             "std_dev_str": "00:00:00.000"
//!           }
//!         ]
//!       ]
//!     }
//!   }
//! }
//! ```
//! </details>
//!
//! <details>
//!  <summary>Benchmark Workloads Series</summary>
//!
//! Benchmark name: benchmark-workload-2
//! ```csv
//! point,ramp_up,repeat,min_sec,max_sec,median_sec,std_dev_sec
//! 1,1,2,0.00063575,0.000637376,0.000636563,0.0000011497556262093261
//! 2,1,2,0.001265333,0.001269584,0.001267458,0.0000030059109268240135
//! 3,1,2,0.001890958,0.001892333,0.001891645,0.0000009722718241315028
//! 4,1,2,0.002512042,0.002604791,0.002558416,0.0000655834468482711
//! 5,1,2,0.003129084,0.00314525,0.003137167,0.000011431088224661727
//! 6,1,2,0.003092583,0.003767833,0.003430208,0.0004774738539962162
//! 7,1,2,0.004055333,0.004084166,0.004069749,0.000020388009821951726
//! 8,1,2,0.004171542,0.005756541,0.004964041,0.0011207635410738967
//! 9,1,2,0.004549334,0.005149792,0.004849563,0.0004245879236177119
//! 10,1,2,0.006351,0.007404083,0.006877541,0.000744642130452273
//! ```
//!
//! Benchmark name: benchmark-workload-1
//! ```csv
//! point,ramp_up,repeat,min_sec,max_sec,median_sec,std_dev_sec
//! 1,1,2,0.001265417,0.001266709,0.001266063,0.0000009135819612930194
//! 2,1,2,0.002518501,0.002534999,0.00252675,0.000011665847676015662
//! 3,1,2,0.003762958,0.004072459,0.003917708,0.00021885025588401765
//! 4,1,2,0.004471833,0.004559375,0.004515604,0.00006190154183863275
//! 5,1,2,0.005676917,0.005765751,0.005721334,0.00006281512379992577
//! 6,1,2,0.006610875,0.007539833,0.007075354,0.0006568725012374928
//! 7,1,2,0.007831792,0.008021959,0.007926875,0.0001344683752579022
//! 8,1,2,0.009668583,0.009832959,0.009750771,0.00011623138426431993
//! 9,1,2,0.009677333,0.011178501,0.010427917,0.0010614860725002473
//! 10,1,2,0.011440417,0.012526,0.011983208,0.0007676231008408358
//! ```
//!
//! </details>
//!
//!
//! <details>
//!  <summary>Find Regressions Example</summary>
//!
//! ```
//! use std::collections::BTreeMap;
//! use std::fmt::{Display, Formatter};
//! use std::thread;
//! use std::time::Duration;
//!
//! use benchmark_rs::benchmarks::Benchmarks;
//! use benchmark_rs::stopwatch::StopWatch;
//! use rand::Rng;
//!
//! #[derive(Clone)]
//! struct Config {
//!     // simulate available resources - CPU cores, memory buffers, etc.
//!     pub resources: u32,
//!     pub workloads: BTreeMap<u64, Duration>,
//! }
//!
//! impl Display for Config {
//!     fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
//!         for k in self.workloads.keys() {
//!             write!(f, "{k} ")?;
//!         }
//!         Ok(())
//!     }
//! }
//!
//! fn example(_stop_watch: &mut StopWatch, config: Config, work: u64) -> Result<(), anyhow::Error> {
//!     let sleep_time = config.workloads.get(&work).unwrap().clone();
//!     thread::sleep(sleep_time / config.resources);
//!     Ok(())
//! }
//!
//! fn modified_example(
//!     _stop_watch: &mut StopWatch,
//!     config: Config,
//!     work: u64,
//! ) -> Result<(), anyhow::Error> {
//!     // we introduce a random deviation in modified example to simulate
//!     // regression introduced by a code change
//!     let deviation = Duration::from_millis(rand::thread_rng().gen_range(0..10));
//!     let sleep_time = config.workloads.get(&work).unwrap().clone();
//!     thread::sleep(sleep_time / config.resources + deviation);
//!     Ok(())
//! }
//!
//! fn main() -> Result<(), anyhow::Error> {
//!     let mut previous_benchmarks = Benchmarks::new("benchmarks");
//!     let workloads: BTreeMap<u64, Duration> = (1..=10)
//!         .map(|i| (i, Duration::from_millis(i * 25)))
//!         .collect();
//!     previous_benchmarks.add(
//!         "benchmark-1",
//!         example,
//!         Config {
//!             resources: 1,
//!             workloads: workloads.clone(),
//!         },
//!         (1..=10).collect(),
//!         5,
//!         3,
//!     )?;
//!     previous_benchmarks.run()?;
//!     let previous_summary = previous_benchmarks.summary_as_json();
//!
//!     let mut current_benchmarks = Benchmarks::new("benchmarks");
//!     current_benchmarks.add(
//!         "benchmark-1",
//!         modified_example,
//!         Config {
//!             resources: 1,
//!             workloads: workloads.clone(),
//!         },
//!         (1..=10).collect(),
//!         5,
//!         3,
//!     )?;
//!     current_benchmarks.add(
//!         "benchmark-2",
//!         modified_example,
//!         Config {
//!             resources: 2,
//!             workloads: workloads.clone(),
//!         },
//!         (1..=10).collect(),
//!         5,
//!         3,
//!     )?;
//!     current_benchmarks.run()?;
//!
//!     // compare results of this run with the results of previous runs with threshold of 5 percent
//!     let analysis_result = current_benchmarks.analyze(Some(previous_summary), 5.0)?;
//!     println!("Analysis result:");
//!     println!("{}", analysis_result.to_string());
//!     assert!(!analysis_result.divergent_series().is_empty());
//!     Ok(())
//! }
//! ```
//!
//! </details>
//!
//!
//! <details>
//!  <summary>Find Regressions Output</summary>
//!  
//! Analysis result. Current and previous values are in nanosecond units, the change is in percents.
//! ```json
//!   {
//!     "name": "benchmarks",
//!     "new_series": [
//!       "benchmark-2"
//!     ],
//!     "equal_series": {},
//!     "divergent_series": {
//!       "benchmark-1": {
//!         "8": {
//!           "Equal": {
//!             "point": "8",
//!             "previous": 200686042,
//!             "current": 206411375,
//!             "change": 2.852880520709064
//!           }
//!         },
//!         "7": {
//!           "Equal": {
//!             "point": "7",
//!             "previous": 176700166,
//!             "current": 183395333,
//!             "change": 3.788998704166474
//!           }
//!         },
//!         "10": {
//!           "Equal": {
//!             "point": "10",
//!             "previous": 251701083,
//!             "current": 256856000,
//!             "change": 2.0480313149864315
//!           }
//!         },
//!         "2": {
//!           "Equal": {
//!             "point": "2",
//!             "previous": 52684667,
//!             "current": 55052334,
//!             "change": 4.494034288951653
//!           }
//!         },
//!         "5": {
//!           "Equal": {
//!             "point": "5",
//!             "previous": 127510583,
//!             "current": 130817709,
//!             "change": 2.5936090340046434
//!           }
//!         },
//!         "6": {
//!           "Equal": {
//!             "point": "6",
//!             "previous": 152803084,
//!             "current": 158141750,
//!             "change": 3.4938208446106955
//!           }
//!         },
//!         "9": {
//!           "Equal": {
//!             "point": "9",
//!             "previous": 225706250,
//!             "current": 229522083,
//!             "change": 1.6906191122310474
//!           }
//!         },
//!         "1": {
//!           "Greater": {
//!             "point": "1",
//!             "previous": 26413209,
//!             "current": 35264166,
//!             "change": 33.50958605597674
//!           }
//!         },
//!         "4": {
//!           "Equal": {
//!             "point": "4",
//!             "previous": 104201208,
//!             "current": 109166958,
//!             "change": 4.7655397622645665
//!           }
//!         },
//!         "3": {
//!           "Equal": {
//!             "point": "3",
//!             "previous": 79801875,
//!             "current": 84060834,
//!             "change": 5.336915955922095
//!           }
//!         }
//!       }
//!     }
//!   }
//! ```
//! </details>
//!
//! # Similar Projects
//! * [criterion](https://crates.io/crates/criterion)
//! * [iai](https://crates.io/crates/iai)
//!
//!
pub mod analysis_result;
pub mod benchmark;
pub mod benchmark_comparison;
pub mod benchmarks;
pub mod disk_usage;
pub mod run_summary;
pub mod series_summary;
pub mod stopwatch;
pub mod summary;