What is benchie?
benchie is a simple-to-use CLI tool that benchmarks performance of your processes and organizes the benchmarking results for you.
Installation
benchie ships as a single executable with no dependencies. You can install it using the installers below, or download a release binary from the releases page.
Shell (Mac, Linux):
curl -fsSL https://raw.githubusercontent.com/benchie-io/benchie/main/scripts/install.sh | sh
PowerShell (Windows):
iwr https://raw.githubusercontent.com/benchie-io/benchie/main/scripts/install.ps1 -useb | iex
Build from Source
benchie can be built and tested on all major platforms. First install Rust from https://rustup.rs and add it to your path.
Install via Cargo
Install the latest version of benchie using Rust's built-in package manager:
Install via Github
- Clone this repository
- Test you toolchain setup by compiling benchie:
- Run tests:
Usage
Benchmark
In order to benchmark (measure runtime similar to the unix-tool time) a process, invocate benchie as follows:
where <COMMAND> can be any command including command-line arguments.
benchie stores additionally some meta information to the benchmark such as information of the commit, timestamps, exit status of the executable, etc.
Moreover, one can pass various options to the benchmarking process.
Tagging
A tag helps to identify benchmarking results. One can pass multiple tags as key-value pairs for each benchmark using --tag key=value as an option.
For instance,
runs the executable ./bubblesort and tags the benchmark with algorithm=bubblesort.
Output Tagging
Tags can also be provided on stdout of the executable at runtime. For instance, benchie will parse the output and if
@benchie key=value
is printed to stdout, the tag key=value is stored as tag to the benchmark.
Display Benchmarking Results
To get an overview of all your benchmarks, simply type
A sample output could be:
+-----------------+-------------+----------------------------------------------------------------+
| key | occurrences | example values |
+-----------------+-------------+----------------------------------------------------------------+
| commit_id | 2 | 014fbb4b2e5c5cc1de7266a708fa909df9915011 |
+-----------------+-------------+----------------------------------------------------------------+
| user_time | 2 | 567µs, 572µs |
+-----------------+-------------+----------------------------------------------------------------+
| algorithm | 2 | mergesort, bubblesort |
+-----------------+-------------+----------------------------------------------------------------+
| created_at | 2 | 2022-05-11 09:11:54.315066 UTC, 2022-05-11 09:11:47.635744 UTC |
+-----------------+-------------+----------------------------------------------------------------+
| status_code | 2 | 0 |
+-----------------+-------------+----------------------------------------------------------------+
which means that we have 2 stored benchmarks, where we tagged one with algorithm=mergesort and the other with algorithm=bubblesort.
Table View
To usefully show benchmarking results, benchie provides a one-dimensional or two-dimensional table view.
For the one-dimensional view, we must provide the --row option and a metric that we want to display.
Both, the row and the metric must occur as a key in at least one benchmark.
To identify occurring keys, use benchie show.
As an example, we use algorithm as row and user_time as metric:
which may give the following output:
+------------+-----------+
| algorithm | user_time |
+------------+-----------+
| bubblesort | 572µs |
+------------+-----------+
| mergesort | 567µs |
+------------+-----------+
For the two-dimensional table view, we need also to pass the --col option, which again must be an occuring key.
Let's assume we made 4 benchmarks: 2 for bubblesort and 2 for mergesort with 100 and 1000 elements, respectively.
Hence, we tagged each benchmark with two tags: algorithm=bubblesort and elements=100, etc.
To show a two-dimensional table view on algorithm and elements with user_time as metric, we can type
which may give the following output:
+------------+----------------+
| | 100 | 1000 |
+------------+----------------+
| bubblesort | 572µs | 944µs |
+------------+----------------+
| mergesort | 567µs | 598µs |
+------------+----------------+
Filtering
To filter the benchmark results, one can pass an equality filter as option.
The syntax for the filter option is --filter key=value, for instance
shows a one-dimensional table view, which lists only entries where algorithm is equal to mergesort.
Contribution
TBA
License
Licensed under the MIT license.