# 🦀 crabz
<p align="center">
<a href="https://github.com/sstadick/crabz/actions?query=workflow%3ACheck"><img src="https://github.com/sstadick/crabz/workflows/Check/badge.svg" alt="Build Status"></a>
<img src="https://img.shields.io/crates/l/crabz.svg" alt="license">
<a href="https://crates.io/crates/crabz"><img src="https://img.shields.io/crates/v/crabz.svg?colorB=319e8c" alt="Version info"></a><br>
Like pigz, but rust.
</p>
A cross platform, fast, compression and decompression tool.
## Synopsis
This is currently a proof of concept CLI tool using the [`gzp`](https://github.com/sstadick/gzp/) crate.
Supported formats:
- Gzip
- Zlib
- Mgzip
- BGZF
- Raw Deflate
- Snap
## Install
* Homebrew / Linuxbrew
```
brew tap sstadick/crabz
brew install crabz
```
* Debian (Ubuntu)
```
curl -LO https://github.com/sstadick/crabz/releases/download/<latest>/crabz-linux-amd64.deb
sudo dpkg -i crabz-linux-amd64.deb
```
* Cargo
```
cargo install crabz
```
* Conda
```
conda install -c conda-forge crabz
```
## Usage
```
❯ crabz -h
Compress and decompress files
USAGE:
crabz [FLAGS] [OPTIONS] [FILE]
FLAGS:
-d, --decompress
Flag to switch to decompressing inputs. Note: this flag may change in future releases
-h, --help
Prints help information
-I, --in-place
Perform the compression / decompression in place.
**NOTE** this will remove the input file at completion.
-V, --version
Prints version information
OPTIONS:
-l, --compression-level <compression-level>
Compression level [default: 6]
-p, --compression-threads <compression-threads>
Number of compression threads to use, or if decompressing a format that allow for multi-threaded
decompression, the number to use. Note that > 4 threads for decompression doesn't seem to help [default:
32]
-f, --format <format>
The format to use [default: gzip] [possible values: gzip, bgzf, mgzip,
zlib, deflate, snap]
-o, --output <output>
Output path to write to, empty or "-" to write to stdout
-P, --pin-at <pin-at>
Specify the physical core to pin threads at.
This can provide a significant performance improvement, but has the downside of possibly conflicting with
other pinned cores. If you are running multiple instances of `crabz` at once you can manually space out the
pinned cores.
# Example
- Instance 1 has `-p 4 -P 0` set indicating that it will use 4 cores pinned at 0, 1, 2, 3
- Instance 2 has `-p 4 -P 4` set indicating that it will use 4 cores pinned at 4, 5, 6, 7
ARGS:
<FILE>
Input file to read from, empty or "-" to read from stdin
```
## Benchmarks
These benchmarks use the data in `bench-data` catted together 100 times. Run with `bash ./benchmark.sh data.txt`.
Benchmark system specs: Ubuntu 20 AMD Ryzen 9 3950X 16-Core Processor w/ 64 GB DDR4 memory and 1TB NVMe Drive
`pigz` v2.4 installed via apt on Ubuntu
Takeaways:
- `crabz` with `zlib` backend is pretty much identical to `pigz`
- `crabz` with `zlib-ng` backend is roughly 30-50% faster than `pigz`
- `crabz` with `rust` backend is roughly 5-10% faster than `pigz`
It is already known that `zlib-ng` is faster than `zlib`, so none of this is groundbreaking. However, I think `crabz` gets an
an edge due to the following:
- `crabz` with `deflate_rust` backend is using all Rust only code, which is in theory more secure / safe.
- `crabz` with `zlib-ng` is easier to install than `pigz` with a `zlib-ng` backend
- `crabz` supports more formats than `pigz`
- `crabz` is cross platform and can run on windows
With regards to block formats like Mgzip and BGZF, `crabz` is using `libdeflater` by default which excels at compressing and
decompression known-sized blocks. This makes block compression formats very fast at a small loss to the compression ratio.
See end of benchmarks section for comparison against `bgzip`.
As `crabz` is just a wrapper for the `gzp` library, the most exciting thing about these benchmarks is that `gzp` is on par with
best in class CLI tools for multi-threaded compression and decompression as a library.
### Flate2 zlib-ng backend
#### Compression
| `crabz -p 1 -c 3 < ./data.txt` | 6.450 ± 0.069 | 6.328 | 6.540 | 16.86 ± 0.24 |
| `pigz -p 1 -3 < ./data.txt` | 11.404 ± 0.152 | 11.186 | 11.717 | 29.81 ± 0.49 |
| `crabz -p 2 -c 3 < ./data.txt` | 3.437 ± 0.017 | 3.418 | 3.461 | 8.98 ± 0.10 |
| `pigz -p 2 -3 < ./data.txt` | 5.868 ± 0.031 | 5.826 | 5.927 | 15.34 ± 0.17 |
| `crabz -p 4 -c 3 < ./data.txt` | 1.741 ± 0.008 | 1.729 | 1.752 | 4.55 ± 0.05 |
| `pigz -p 4 -3 < ./data.txt` | 2.952 ± 0.008 | 2.939 | 2.960 | 7.72 ± 0.08 |
| `crabz -p 8 -c 3 < ./data.txt` | 0.889 ± 0.004 | 0.882 | 0.895 | 2.32 ± 0.02 |
| `pigz -p 8 -3 < ./data.txt` | 1.505 ± 0.008 | 1.493 | 1.520 | 3.93 ± 0.04 |
| `crabz -p 16 -c 3 < ./data.txt` | 0.485 ± 0.014 | 0.457 | 0.502 | 1.27 ± 0.04 |
| `pigz -p 16 -3 < ./data.txt` | 0.775 ± 0.011 | 0.764 | 0.797 | 2.02 ± 0.04 |
| `crabz -p 32 -c 3 < ./data.txt` | 0.383 ± 0.004 | 0.375 | 0.388 | 1.00 |
| `pigz -p 32 -3 < ./data.txt` | 0.699 ± 0.029 | 0.668 | 0.770 | 1.83 ± 0.08 |
| `crabz -p 1 -c 6 < ./data.txt` | 10.367 ± 0.211 | 10.106 | 10.642 | 27.10 ± 0.61 |
| `pigz -p 1 -6 < ./data.txt` | 26.734 ± 0.345 | 26.234 | 27.135 | 69.89 ± 1.12 |
| `crabz -p 2 -c 6 < ./data.txt` | 5.366 ± 0.036 | 5.299 | 5.429 | 14.03 ± 0.16 |
| `pigz -p 2 -6 < ./data.txt` | 13.589 ± 0.083 | 13.428 | 13.679 | 35.52 ± 0.40 |
| `crabz -p 4 -c 6 < ./data.txt` | 2.719 ± 0.021 | 2.694 | 2.757 | 7.11 ± 0.09 |
| `pigz -p 4 -6 < ./data.txt` | 6.887 ± 0.013 | 6.871 | 6.916 | 18.00 ± 0.17 |
| `crabz -p 8 -c 6 < ./data.txt` | 1.381 ± 0.007 | 1.372 | 1.397 | 3.61 ± 0.04 |
| `pigz -p 8 -6 < ./data.txt` | 3.479 ± 0.008 | 3.463 | 3.488 | 9.09 ± 0.09 |
| `crabz -p 16 -c 6 < ./data.txt` | 0.745 ± 0.022 | 0.727 | 0.804 | 1.95 ± 0.06 |
| `pigz -p 16 -6 < ./data.txt` | 1.818 ± 0.036 | 1.765 | 1.874 | 4.75 ± 0.10 |
| `crabz -p 32 -c 6 < ./data.txt` | 0.549 ± 0.006 | 0.538 | 0.557 | 1.44 ± 0.02 |
| `pigz -p 32 -6 < ./data.txt` | 1.187 ± 0.011 | 1.172 | 1.210 | 3.10 ± 0.04 |
| `crabz -p 1 -c 9 < ./data.txt` | 30.114 ± 0.196 | 29.842 | 30.420 | 78.72 ± 0.90 |
| `pigz -p 1 -9 < ./data.txt` | 51.369 ± 0.164 | 51.246 | 51.698 | 134.29 ± 1.33 |
| `crabz -p 2 -c 9 < ./data.txt` | 15.371 ± 0.070 | 15.202 | 15.443 | 40.18 ± 0.42 |
| `pigz -p 2 -9 < ./data.txt` | 26.452 ± 0.085 | 26.253 | 26.576 | 69.15 ± 0.69 |
| `crabz -p 4 -c 9 < ./data.txt` | 7.729 ± 0.022 | 7.699 | 7.768 | 20.20 ± 0.20 |
| `pigz -p 4 -9 < ./data.txt` | 13.365 ± 0.047 | 13.271 | 13.449 | 34.94 ± 0.35 |
| `crabz -p 8 -c 9 < ./data.txt` | 3.901 ± 0.006 | 3.889 | 3.910 | 10.20 ± 0.10 |
| `pigz -p 8 -9 < ./data.txt` | 6.749 ± 0.014 | 6.737 | 6.781 | 17.64 ± 0.17 |
| `crabz -p 16 -c 9 < ./data.txt` | 2.039 ± 0.024 | 1.997 | 2.071 | 5.33 ± 0.08 |
| `pigz -p 16 -9 < ./data.txt` | 3.486 ± 0.054 | 3.426 | 3.574 | 9.11 ± 0.17 |
| `crabz -p 32 -c 9 < ./data.txt` | 1.337 ± 0.072 | 1.220 | 1.411 | 3.49 ± 0.19 |
| `pigz -p 32 -9 < ./data.txt` | 2.203 ± 0.114 | 2.082 | 2.378 | 5.76 ± 0.30 |
#### Decompression
| `crabz -d < ./data.3.txt.gz` | 1.422 ± 0.010 | 1.411 | 1.437 | 1.03 ± 0.02 |
| `pigz -d < ./data.3.txt.gz` | 1.674 ± 0.031 | 1.621 | 1.705 | 1.21 ± 0.03 |
| `crabz -d < ./data.6.txt.gz` | 1.403 ± 0.016 | 1.389 | 1.427 | 1.01 ± 0.02 |
| `pigz -d < ./data.6.txt.gz` | 1.724 ± 0.026 | 1.697 | 1.766 | 1.24 ± 0.02 |
| `crabz -d < ./data.9.txt.gz` | 1.385 ± 0.018 | 1.359 | 1.416 | 1.00 |
| `pigz -d < ./data.9.txt.gz` | 1.745 ± 0.044 | 1.684 | 1.797 | 1.26 ± 0.04 |
### Flate2 zlib backend
#### Compression
| `crabz -p 1 -c 3 < ./data.txt` | 11.248 ± 0.247 | 11.085 | 11.532 | 20.23 ± 0.45 |
| `pigz -p 1 -3 < ./data.txt` | 11.296 ± 0.171 | 11.104 | 11.434 | 20.32 ± 0.31 |
| `crabz -p 2 -c 3 < ./data.txt` | 5.681 ± 0.040 | 5.645 | 5.725 | 10.22 ± 0.08 |
| `pigz -p 2 -3 < ./data.txt` | 5.926 ± 0.015 | 5.916 | 5.944 | 10.66 ± 0.04 |
| `crabz -p 4 -c 3 < ./data.txt` | 2.891 ± 0.007 | 2.883 | 2.895 | 5.20 ± 0.02 |
| `pigz -p 4 -3 < ./data.txt` | 2.966 ± 0.013 | 2.955 | 2.980 | 5.34 ± 0.03 |
| `crabz -p 8 -c 3 < ./data.txt` | 1.461 ± 0.003 | 1.459 | 1.465 | 2.63 ± 0.01 |
| `pigz -p 8 -3 < ./data.txt` | 1.509 ± 0.004 | 1.505 | 1.512 | 2.71 ± 0.01 |
| `crabz -p 16 -c 3 < ./data.txt` | 0.784 ± 0.010 | 0.775 | 0.795 | 1.41 ± 0.02 |
| `pigz -p 16 -3 < ./data.txt` | 0.772 ± 0.010 | 0.765 | 0.784 | 1.39 ± 0.02 |
| `crabz -p 32 -c 3 < ./data.txt` | 0.556 ± 0.002 | 0.554 | 0.557 | 1.00 |
| `pigz -p 32 -3 < ./data.txt` | 0.743 ± 0.047 | 0.694 | 0.786 | 1.34 ± 0.08 |
| `crabz -p 1 -c 6 < ./data.txt` | 26.366 ± 0.154 | 26.189 | 26.469 | 47.42 ± 0.31 |
| `pigz -p 1 -6 < ./data.txt` | 26.688 ± 0.103 | 26.579 | 26.783 | 48.00 ± 0.23 |
| `crabz -p 2 -c 6 < ./data.txt` | 13.443 ± 0.069 | 13.400 | 13.523 | 24.18 ± 0.14 |
| `pigz -p 2 -6 < ./data.txt` | 13.605 ± 0.059 | 13.567 | 13.673 | 24.47 ± 0.13 |
| `crabz -p 4 -c 6 < ./data.txt` | 6.833 ± 0.005 | 6.828 | 6.837 | 12.29 ± 0.03 |
| `pigz -p 4 -6 < ./data.txt` | 6.866 ± 0.028 | 6.834 | 6.884 | 12.35 ± 0.06 |
| `crabz -p 8 -c 6 < ./data.txt` | 3.446 ± 0.000 | 3.445 | 3.446 | 6.20 ± 0.02 |
| `pigz -p 8 -6 < ./data.txt` | 3.482 ± 0.002 | 3.480 | 3.483 | 6.26 ± 0.02 |
| `crabz -p 16 -c 6 < ./data.txt` | 1.822 ± 0.012 | 1.813 | 1.835 | 3.28 ± 0.02 |
| `pigz -p 16 -6 < ./data.txt` | 1.771 ± 0.004 | 1.767 | 1.776 | 3.19 ± 0.01 |
| `crabz -p 32 -c 6 < ./data.txt` | 1.178 ± 0.008 | 1.171 | 1.187 | 2.12 ± 0.02 |
| `pigz -p 32 -6 < ./data.txt` | 1.184 ± 0.001 | 1.184 | 1.185 | 2.13 ± 0.01 |
| `crabz -p 1 -c 9 < ./data.txt` | 52.122 ± 0.288 | 51.790 | 52.293 | 93.75 ± 0.58 |
| `pigz -p 1 -9 < ./data.txt` | 53.031 ± 0.071 | 52.951 | 53.085 | 95.39 ± 0.29 |
| `crabz -p 2 -c 9 < ./data.txt` | 26.287 ± 0.047 | 26.249 | 26.339 | 47.28 ± 0.15 |
| `pigz -p 2 -9 < ./data.txt` | 26.409 ± 0.238 | 26.190 | 26.662 | 47.50 ± 0.45 |
| `crabz -p 4 -c 9 < ./data.txt` | 13.373 ± 0.051 | 13.317 | 13.419 | 24.05 ± 0.11 |
| `pigz -p 4 -9 < ./data.txt` | 13.414 ± 0.035 | 13.383 | 13.451 | 24.13 ± 0.09 |
| `crabz -p 8 -c 9 < ./data.txt` | 6.733 ± 0.003 | 6.731 | 6.736 | 12.11 ± 0.03 |
| `pigz -p 8 -9 < ./data.txt` | 6.763 ± 0.004 | 6.761 | 6.767 | 12.16 ± 0.03 |
| `crabz -p 16 -c 9 < ./data.txt` | 3.487 ± 0.034 | 3.450 | 3.517 | 6.27 ± 0.06 |
| `pigz -p 16 -9 < ./data.txt` | 3.459 ± 0.021 | 3.434 | 3.473 | 6.22 ± 0.04 |
| `crabz -p 32 -c 9 < ./data.txt` | 2.088 ± 0.008 | 2.081 | 2.097 | 3.76 ± 0.02 |
| `pigz -p 32 -9 < ./data.txt` | 2.107 ± 0.023 | 2.090 | 2.133 | 3.79 ± 0.04 |
#### Decompression
### Flate2 rust backend
#### Compression
| `crabz -p 1 -c 3 < ./data.txt` | 10.167 ± 0.164 | 10.050 | 10.355 | 18.57 ± 0.33 |
| `pigz -p 1 -3 < ./data.txt` | 11.338 ± 0.071 | 11.292 | 11.420 | 20.71 ± 0.21 |
| `crabz -p 2 -c 3 < ./data.txt` | 4.912 ± 0.013 | 4.898 | 4.920 | 8.97 ± 0.08 |
| `pigz -p 2 -3 < ./data.txt` | 5.876 ± 0.047 | 5.826 | 5.919 | 10.73 ± 0.12 |
| `crabz -p 4 -c 3 < ./data.txt` | 2.463 ± 0.018 | 2.447 | 2.482 | 4.50 ± 0.05 |
| `pigz -p 4 -3 < ./data.txt` | 2.967 ± 0.008 | 2.958 | 2.972 | 5.42 ± 0.05 |
| `crabz -p 8 -c 3 < ./data.txt` | 1.255 ± 0.005 | 1.250 | 1.261 | 2.29 ± 0.02 |
| `pigz -p 8 -3 < ./data.txt` | 1.509 ± 0.002 | 1.507 | 1.511 | 2.76 ± 0.02 |
| `crabz -p 16 -c 3 < ./data.txt` | 0.705 ± 0.030 | 0.673 | 0.731 | 1.29 ± 0.05 |
| `pigz -p 16 -3 < ./data.txt` | 0.780 ± 0.015 | 0.768 | 0.797 | 1.42 ± 0.03 |
| `crabz -p 32 -c 3 < ./data.txt` | 0.547 ± 0.004 | 0.544 | 0.552 | 1.00 |
| `pigz -p 32 -3 < ./data.txt` | 0.755 ± 0.025 | 0.726 | 0.771 | 1.38 ± 0.05 |
| `crabz -p 1 -c 6 < ./data.txt` | 27.064 ± 0.288 | 26.863 | 27.394 | 49.44 ± 0.66 |
| `pigz -p 1 -6 < ./data.txt` | 27.034 ± 0.090 | 26.938 | 27.117 | 49.38 ± 0.43 |
| `crabz -p 2 -c 6 < ./data.txt` | 12.400 ± 0.083 | 12.321 | 12.487 | 22.65 ± 0.24 |
| `pigz -p 2 -6 < ./data.txt` | 13.619 ± 0.074 | 13.558 | 13.702 | 24.88 ± 0.24 |
| `crabz -p 4 -c 6 < ./data.txt` | 6.279 ± 0.023 | 6.263 | 6.305 | 11.47 ± 0.10 |
| `pigz -p 4 -6 < ./data.txt` | 6.879 ± 0.020 | 6.867 | 6.901 | 12.57 ± 0.11 |
| `crabz -p 8 -c 6 < ./data.txt` | 3.189 ± 0.010 | 3.178 | 3.198 | 5.83 ± 0.05 |
| `pigz -p 8 -6 < ./data.txt` | 3.477 ± 0.007 | 3.470 | 3.483 | 6.35 ± 0.05 |
| `crabz -p 16 -c 6 < ./data.txt` | 1.756 ± 0.015 | 1.740 | 1.771 | 3.21 ± 0.04 |
| `pigz -p 16 -6 < ./data.txt` | 1.799 ± 0.024 | 1.779 | 1.827 | 3.29 ± 0.05 |
| `crabz -p 32 -c 6 < ./data.txt` | 1.192 ± 0.011 | 1.183 | 1.205 | 2.18 ± 0.03 |
| `pigz -p 32 -6 < ./data.txt` | 1.196 ± 0.016 | 1.183 | 1.214 | 2.19 ± 0.03 |
| `crabz -p 1 -c 9 < ./data.txt` | 44.907 ± 0.283 | 44.585 | 45.116 | 82.03 ± 0.84 |
| `pigz -p 1 -9 < ./data.txt` | 53.109 ± 1.049 | 52.373 | 54.311 | 97.02 ± 2.07 |
| `crabz -p 2 -c 9 < ./data.txt` | 19.977 ± 0.159 | 19.819 | 20.136 | 36.49 ± 0.41 |
| `pigz -p 2 -9 < ./data.txt` | 26.562 ± 0.134 | 26.407 | 26.643 | 48.52 ± 0.46 |
| `crabz -p 4 -c 9 < ./data.txt` | 10.397 ± 0.484 | 10.070 | 10.953 | 18.99 ± 0.90 |
| `pigz -p 4 -9 < ./data.txt` | 13.346 ± 0.040 | 13.300 | 13.372 | 24.38 ± 0.21 |
| `crabz -p 8 -c 9 < ./data.txt` | 5.100 ± 0.021 | 5.076 | 5.114 | 9.32 ± 0.08 |
| `pigz -p 8 -9 < ./data.txt` | 6.754 ± 0.016 | 6.736 | 6.767 | 12.34 ± 0.10 |
| `crabz -p 16 -c 9 < ./data.txt` | 2.716 ± 0.014 | 2.708 | 2.732 | 4.96 ± 0.05 |
| `pigz -p 16 -9 < ./data.txt` | 3.444 ± 0.038 | 3.420 | 3.487 | 6.29 ± 0.09 |
| `crabz -p 32 -c 9 < ./data.txt` | 1.747 ± 0.009 | 1.740 | 1.758 | 3.19 ± 0.03 |
| `pigz -p 32 -9 < ./data.txt` | 2.086 ± 0.008 | 2.077 | 2.093 | 3.81 ± 0.03 |
#### Decompression
| `crabz -d < ./data.3.txt.gz` | 1.599 ± 0.014 | 1.573 | 1.615 | 1.00 |
| `pigz -d < ./data.3.txt.gz` | 1.696 ± 0.020 | 1.654 | 1.725 | 1.06 ± 0.02 |
| `crabz -d < ./data.6.txt.gz` | 1.615 ± 0.012 | 1.586 | 1.626 | 1.01 ± 0.01 |
| `pigz -d < ./data.6.txt.gz` | 1.760 ± 0.030 | 1.687 | 1.797 | 1.10 ± 0.02 |
| `crabz -d < ./data.9.txt.gz` | 1.613 ± 0.014 | 1.596 | 1.641 | 1.01 ± 0.01 |
| `pigz -d < ./data.9.txt.gz` | 1.767 ± 0.012 | 1.748 | 1.787 | 1.11 ± 0.01 |
### Block Formats with libdeflater
#### Decompression
| `crabz -p 1 -d -f mgzip ./bdata.3.txt.gz > data.txt` | 1.221 ± 0.164 | 1.073 | 1.397 | 2.32 ± 0.31 |
| `pigz -d -c ./bdata.3.txt.gz > data.txt` | 2.415 ± 0.063 | 2.347 | 2.472 | 4.58 ± 0.14 |
| `crabz -p 1 -d -f mgzip ./bdata.6.txt.gz > data.txt` | 1.256 ± 0.063 | 1.200 | 1.325 | 2.38 ± 0.13 |
| `pigz -d -c ./bdata.6.txt.gz > data.txt` | 2.513 ± 0.052 | 2.467 | 2.569 | 4.77 ± 0.13 |
| `crabz -p 1 -d -f mgzip ./bdata.9.txt.gz > data.txt` | 1.147 ± 0.065 | 1.094 | 1.219 | 2.18 ± 0.13 |
| `pigz -d -c ./bdata.9.txt.gz > data.txt` | 2.394 ± 0.118 | 2.262 | 2.488 | 4.54 ± 0.24 |
| `crabz -p 1 -d -f mgzip ./bdata.12.txt.gz > data.txt` | 1.165 ± 0.074 | 1.106 | 1.248 | 2.21 ± 0.15 |
| `pigz -d -c ./bdata.12.txt.gz > data.txt` | 2.457 ± 0.067 | 2.408 | 2.534 | 4.66 ± 0.15 |
| `crabz -p 2 -d -f mgzip ./bdata.3.txt.gz > data.txt` | 0.634 ± 0.008 | 0.628 | 0.642 | 1.20 ± 0.03 |
| `pigz -d -c ./bdata.3.txt.gz > data.txt` | 2.379 ± 0.012 | 2.368 | 2.391 | 4.51 ± 0.08 |
| `crabz -p 2 -d -f mgzip ./bdata.6.txt.gz > data.txt` | 0.645 ± 0.015 | 0.629 | 0.658 | 1.22 ± 0.03 |
| `pigz -d -c ./bdata.6.txt.gz > data.txt` | 2.438 ± 0.073 | 2.356 | 2.497 | 4.62 ± 0.16 |
| `crabz -p 2 -d -f mgzip ./bdata.9.txt.gz > data.txt` | 0.659 ± 0.015 | 0.644 | 0.674 | 1.25 ± 0.04 |
| `pigz -d -c ./bdata.9.txt.gz > data.txt` | 2.451 ± 0.075 | 2.400 | 2.538 | 4.65 ± 0.16 |
| `crabz -p 2 -d -f mgzip ./bdata.12.txt.gz > data.txt` | 0.656 ± 0.015 | 0.647 | 0.673 | 1.24 ± 0.04 |
| `pigz -d -c ./bdata.12.txt.gz > data.txt` | 2.450 ± 0.045 | 2.412 | 2.500 | 4.65 ± 0.12 |
| `crabz -p 4 -d -f mgzip ./bdata.3.txt.gz > data.txt` | 0.577 ± 0.024 | 0.554 | 0.603 | 1.10 ± 0.05 |
| `pigz -d -c ./bdata.3.txt.gz > data.txt` | 2.459 ± 0.052 | 2.420 | 2.518 | 4.66 ± 0.13 |
| `crabz -p 4 -d -f mgzip ./bdata.6.txt.gz > data.txt` | 0.559 ± 0.024 | 0.531 | 0.576 | 1.06 ± 0.05 |
| `pigz -d -c ./bdata.6.txt.gz > data.txt` | 2.538 ± 0.044 | 2.502 | 2.587 | 4.81 ± 0.12 |
| `crabz -p 4 -d -f mgzip ./bdata.9.txt.gz > data.txt` | 0.552 ± 0.011 | 0.539 | 0.560 | 1.05 ± 0.03 |
| `pigz -d -c ./bdata.9.txt.gz > data.txt` | 2.402 ± 0.018 | 2.385 | 2.420 | 4.56 ± 0.08 |
| `crabz -p 4 -d -f mgzip ./bdata.12.txt.gz > data.txt` | 0.592 ± 0.040 | 0.546 | 0.616 | 1.12 ± 0.08 |
| `pigz -d -c ./bdata.12.txt.gz > data.txt` | 2.525 ± 0.038 | 2.484 | 2.558 | 4.79 ± 0.11 |
| `crabz -p 8 -d -f mgzip ./bdata.3.txt.gz > data.txt` | 0.563 ± 0.013 | 0.548 | 0.571 | 1.07 ± 0.03 |
| `pigz -d -c ./bdata.3.txt.gz > data.txt` | 2.490 ± 0.126 | 2.369 | 2.621 | 4.72 ± 0.25 |
| `crabz -p 8 -d -f mgzip ./bdata.6.txt.gz > data.txt` | 0.552 ± 0.018 | 0.533 | 0.569 | 1.05 ± 0.04 |
| `pigz -d -c ./bdata.6.txt.gz > data.txt` | 2.531 ± 0.115 | 2.417 | 2.647 | 4.80 ± 0.23 |
| `crabz -p 8 -d -f mgzip ./bdata.9.txt.gz > data.txt` | 0.603 ± 0.029 | 0.583 | 0.636 | 1.14 ± 0.06 |
| `pigz -d -c ./bdata.9.txt.gz > data.txt` | 2.483 ± 0.042 | 2.435 | 2.515 | 4.71 ± 0.11 |
| `crabz -p 8 -d -f mgzip ./bdata.12.txt.gz > data.txt` | 0.527 ± 0.009 | 0.519 | 0.537 | 1.00 |
| `pigz -d -c ./bdata.12.txt.gz > data.txt` | 2.524 ± 0.093 | 2.417 | 2.583 | 4.79 ± 0.19 |
| `crabz -p 16 -d -f mgzip ./bdata.3.txt.gz > data.txt` | 0.603 ± 0.058 | 0.551 | 0.665 | 1.14 ± 0.11 |
| `pigz -d -c ./bdata.3.txt.gz > data.txt` | 2.392 ± 0.007 | 2.384 | 2.397 | 4.54 ± 0.08 |
| `crabz -p 16 -d -f mgzip ./bdata.6.txt.gz > data.txt` | 0.611 ± 0.065 | 0.565 | 0.686 | 1.16 ± 0.13 |
| `pigz -d -c ./bdata.6.txt.gz > data.txt` | 2.593 ± 0.148 | 2.427 | 2.712 | 4.92 ± 0.29 |
| `crabz -p 16 -d -f mgzip ./bdata.9.txt.gz > data.txt` | 0.564 ± 0.027 | 0.541 | 0.594 | 1.07 ± 0.05 |
| `pigz -d -c ./bdata.9.txt.gz > data.txt` | 2.426 ± 0.023 | 2.404 | 2.450 | 4.60 ± 0.09 |
| `crabz -p 16 -d -f mgzip ./bdata.12.txt.gz > data.txt` | 0.601 ± 0.020 | 0.582 | 0.623 | 1.14 ± 0.04 |
| `pigz -d -c ./bdata.12.txt.gz > data.txt` | 2.528 ± 0.022 | 2.507 | 2.550 | 4.80 ± 0.09 |
| `crabz -p 32 -d -f mgzip ./bdata.3.txt.gz > data.txt` | 0.595 ± 0.019 | 0.577 | 0.614 | 1.13 ± 0.04 |
| `pigz -d -c ./bdata.3.txt.gz > data.txt` | 2.544 ± 0.107 | 2.422 | 2.621 | 4.83 ± 0.22 |
| `crabz -p 32 -d -f mgzip ./bdata.6.txt.gz > data.txt` | 0.601 ± 0.021 | 0.586 | 0.626 | 1.14 ± 0.05 |
| `pigz -d -c ./bdata.6.txt.gz > data.txt` | 2.519 ± 0.114 | 2.435 | 2.649 | 4.78 ± 0.23 |
| `crabz -p 32 -d -f mgzip ./bdata.9.txt.gz > data.txt` | 0.565 ± 0.023 | 0.539 | 0.579 | 1.07 ± 0.05 |
| `pigz -d -c ./bdata.9.txt.gz > data.txt` | 2.487 ± 0.064 | 2.415 | 2.540 | 4.72 ± 0.15 |
| `crabz -p 32 -d -f mgzip ./bdata.12.txt.gz > data.txt` | 0.557 ± 0.013 | 0.548 | 0.571 | 1.06 ± 0.03 |
| `pigz -d -c ./bdata.12.txt.gz > data.txt` | 2.505 ± 0.105 | 2.442 | 2.626 | 4.75 ± 0.22 |
#### `crabz`, `pigz`, and `bgzip`
These benchmarks were run on the `all_train.csv` data found [here](https://archive.ics.uci.edu/ml/machine-learning-databases/00347/all_train.csv.gz)
#### Compression
| `crabz -p 2 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz` | 15.837 ± 0.137 | 15.688 | 15.959 | 5.52 ± 0.13 |
| `bgzip -c -@ 2 -l 2 ./data.txt > ./data.out.txt.gz` | 19.471 ± 0.178 | 19.268 | 19.602 | 6.78 ± 0.16 |
| `crabz -p 2 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz` | 19.723 ± 0.632 | 19.285 | 20.448 | 6.87 ± 0.26 |
| `pigz -c -p 2 -2 ./data.txt > ./data.out.txt.gz` | 32.249 ± 0.024 | 32.226 | 32.274 | 11.24 ± 0.24 |
| `crabz -p 4 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz` | 8.601 ± 0.538 | 8.040 | 9.113 | 3.00 ± 0.20 |
| `bgzip -c -@ 4 -l 2 ./data.txt > ./data.out.txt.gz` | 10.953 ± 0.033 | 10.929 | 10.990 | 3.82 ± 0.08 |
| `crabz -p 4 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz` | 10.887 ± 0.584 | 10.236 | 11.364 | 3.79 ± 0.22 |
| `pigz -c -p 4 -2 ./data.txt > ./data.out.txt.gz` | 16.493 ± 0.323 | 16.257 | 16.861 | 5.75 ± 0.17 |
| `crabz -p 8 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz` | 5.206 ± 0.372 | 4.780 | 5.464 | 1.81 ± 0.14 |
| `bgzip -c -@ 8 -l 2 ./data.txt > ./data.out.txt.gz` | 6.920 ± 0.033 | 6.893 | 6.957 | 2.41 ± 0.05 |
| `crabz -p 8 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz` | 5.893 ± 0.135 | 5.777 | 6.041 | 2.05 ± 0.06 |
| `pigz -c -p 8 -2 ./data.txt > ./data.out.txt.gz` | 8.974 ± 0.467 | 8.553 | 9.477 | 3.13 ± 0.18 |
| `crabz -p 16 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz` | 2.870 ± 0.061 | 2.816 | 2.936 | 1.00 |
| `bgzip -c -@ 16 -l 2 ./data.txt > ./data.out.txt.gz` | 5.124 ± 0.107 | 5.040 | 5.244 | 1.79 ± 0.05 |
| `crabz -p 16 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz` | 4.250 ± 0.323 | 3.933 | 4.579 | 1.48 ± 0.12 |
| `pigz -c -p 16 -2 ./data.txt > ./data.out.txt.gz` | 4.767 ± 0.223 | 4.513 | 4.933 | 1.66 ± 0.09 |
| `crabz -p 32 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz` | 3.669 ± 0.303 | 3.320 | 3.865 | 1.28 ± 0.11 |
| `bgzip -c -@ 32 -l 2 ./data.txt > ./data.out.txt.gz` | 4.676 ± 0.038 | 4.632 | 4.701 | 1.63 ± 0.04 |
| `crabz -p 32 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz` | 4.324 ± 0.246 | 4.143 | 4.605 | 1.51 ± 0.09 |
| `pigz -c -p 32 -2 ./data.txt > ./data.out.txt.gz` | 5.854 ± 0.070 | 5.795 | 5.931 | 2.04 ± 0.05 |
| `crabz -p 2 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz` | 27.696 ± 0.147 | 27.593 | 27.864 | 9.65 ± 0.21 |
| `bgzip -c -@ 2 -l 6 ./data.txt > ./data.out.txt.gz` | 30.961 ± 0.446 | 30.446 | 31.231 | 10.79 ± 0.28 |
| `crabz -p 2 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz` | 36.229 ± 0.175 | 36.092 | 36.427 | 12.62 ± 0.27 |
| `pigz -c -p 2 -6 ./data.txt > ./data.out.txt.gz` | 97.175 ± 0.571 | 96.743 | 97.823 | 33.86 ± 0.74 |
| `crabz -p 4 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz` | 14.802 ± 0.436 | 14.316 | 15.159 | 5.16 ± 0.19 |
| `bgzip -c -@ 4 -l 6 ./data.txt > ./data.out.txt.gz` | 16.927 ± 0.130 | 16.789 | 17.048 | 5.90 ± 0.13 |
| `crabz -p 4 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz` | 19.192 ± 0.675 | 18.629 | 19.940 | 6.69 ± 0.27 |
| `pigz -c -p 4 -6 ./data.txt > ./data.out.txt.gz` | 49.305 ± 0.114 | 49.203 | 49.429 | 17.18 ± 0.37 |
| `crabz -p 8 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz` | 7.833 ± 0.065 | 7.784 | 7.907 | 2.73 ± 0.06 |
| `bgzip -c -@ 8 -l 6 ./data.txt > ./data.out.txt.gz` | 9.858 ± 0.105 | 9.739 | 9.939 | 3.43 ± 0.08 |
| `crabz -p 8 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz` | 10.417 ± 0.979 | 9.626 | 11.511 | 3.63 ± 0.35 |
| `pigz -c -p 8 -6 ./data.txt > ./data.out.txt.gz` | 25.276 ± 0.170 | 25.083 | 25.404 | 8.81 ± 0.20 |
| `crabz -p 16 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz` | 4.704 ± 0.321 | 4.337 | 4.937 | 1.64 ± 0.12 |
| `bgzip -c -@ 16 -l 6 ./data.txt > ./data.out.txt.gz` | 6.565 ± 0.155 | 6.429 | 6.734 | 2.29 ± 0.07 |
| `crabz -p 16 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz` | 5.722 ± 0.320 | 5.530 | 6.092 | 1.99 ± 0.12 |
| `pigz -c -p 16 -6 ./data.txt > ./data.out.txt.gz` | 13.673 ± 0.129 | 13.525 | 13.762 | 4.76 ± 0.11 |
| `crabz -p 32 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz` | 4.202 ± 0.213 | 3.957 | 4.328 | 1.46 ± 0.08 |
| `bgzip -c -@ 32 -l 6 ./data.txt > ./data.out.txt.gz` | 5.538 ± 0.135 | 5.395 | 5.663 | 1.93 ± 0.06 |
| `crabz -p 32 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz` | 5.488 ± 0.064 | 5.423 | 5.550 | 1.91 ± 0.05 |
| `pigz -c -p 32 -6 ./data.txt > ./data.out.txt.gz` | 9.079 ± 0.286 | 8.808 | 9.379 | 3.16 ± 0.12 |
| `crabz -p 2 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz` | 162.875 ± 0.100 | 162.778 | 162.977 | 56.75 ± 1.20 |
| `bgzip -c -@ 2 -l 9 ./data.txt > ./data.out.txt.gz` | 172.428 ± 0.242 | 172.207 | 172.687 | 60.08 ± 1.27 |
| `crabz -p 2 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz` | 139.245 ± 0.270 | 138.974 | 139.514 | 48.52 ± 1.03 |
| `pigz -c -p 2 -9 ./data.txt > ./data.out.txt.gz` | 209.645 ± 0.058 | 209.580 | 209.691 | 73.05 ± 1.55 |
| `crabz -p 4 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz` | 84.624 ± 0.185 | 84.414 | 84.762 | 29.49 ± 0.63 |
| `bgzip -c -@ 4 -l 9 ./data.txt > ./data.out.txt.gz` | 87.228 ± 0.232 | 87.053 | 87.492 | 30.39 ± 0.65 |
| `crabz -p 4 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz` | 72.339 ± 0.166 | 72.187 | 72.517 | 25.21 ± 0.54 |
| `pigz -c -p 4 -9 ./data.txt > ./data.out.txt.gz` | 106.579 ± 0.236 | 106.307 | 106.731 | 37.14 ± 0.79 |
| `crabz -p 8 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz` | 42.988 ± 0.130 | 42.905 | 43.138 | 14.98 ± 0.32 |
| `bgzip -c -@ 8 -l 9 ./data.txt > ./data.out.txt.gz` | 44.550 ± 0.097 | 44.449 | 44.642 | 15.52 ± 0.33 |
| `crabz -p 8 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz` | 36.555 ± 0.030 | 36.521 | 36.579 | 12.74 ± 0.27 |
| `pigz -c -p 8 -9 ./data.txt > ./data.out.txt.gz` | 54.047 ± 0.016 | 54.030 | 54.062 | 18.83 ± 0.40 |
| `crabz -p 16 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz` | 22.391 ± 0.234 | 22.154 | 22.623 | 7.80 ± 0.18 |
| `bgzip -c -@ 16 -l 9 ./data.txt > ./data.out.txt.gz` | 24.041 ± 0.237 | 23.813 | 24.286 | 8.38 ± 0.20 |
| `crabz -p 16 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz` | 19.285 ± 0.125 | 19.141 | 19.363 | 6.72 ± 0.15 |
| `pigz -c -p 16 -9 ./data.txt > ./data.out.txt.gz` | 27.645 ± 0.078 | 27.579 | 27.731 | 9.63 ± 0.21 |
| `crabz -p 32 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz` | 15.148 ± 0.138 | 14.992 | 15.252 | 5.28 ± 0.12 |
| `bgzip -c -@ 32 -l 9 ./data.txt > ./data.out.txt.gz` | 16.091 ± 0.193 | 15.874 | 16.243 | 5.61 ± 0.14 |
| `crabz -p 32 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz` | 11.832 ± 0.168 | 11.637 | 11.930 | 4.12 ± 0.11 |
| `pigz -c -p 32 -9 ./data.txt > ./data.out.txt.gz` | 16.912 ± 0.095 | 16.804 | 16.982 | 5.89 ± 0.13 |
#### Decompression
| `crabz -d -p 4 -f bgzf ./data.txt.gz > ./data.out.txt` | 5.941 ± 0.172 | 5.745 | 6.070 | 1.11 ± 0.09 |
| `bgzip -d -c -@ 4 ./data.txt.gz > ./data.out.txt` | 5.357 ± 0.407 | 4.925 | 5.734 | 1.00 |
| `crabz -d -p 8 -f bgzf ./data.txt.gz > ./data.out.txt` | 5.569 ± 0.496 | 5.023 | 5.990 | 1.04 ± 0.12 |
| `bgzip -d -c -@ 8 ./data.txt.gz > ./data.out.txt` | 5.867 ± 0.252 | 5.682 | 6.154 | 1.10 ± 0.10 |
| `crabz -d -p 16 -f bgzf ./data.txt.gz > ./data.out.txt` | 5.663 ± 0.240 | 5.506 | 5.939 | 1.06 ± 0.09 |
| `bgzip -d -c -@ 16 ./data.txt.gz > ./data.out.txt` | 5.534 ± 0.124 | 5.416 | 5.663 | 1.03 ± 0.08 |
## TODOs
- Add some form of auto format detection, even just by file extension