crabz 0.1.3

Parallel Compression
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# 🦀 crabz

Like pigz, but rust.

## Synopsis

This is currently a proof of concept CLI tool using the [`gzp`](https://github.com/sstadick/gzp/) crate.

## Install

```
cargo install crabz
```

## Benchmarks

This are very anecdotal. Data [here](https://archive.ics.uci.edu/ml/machine-learning-databases/00347/all_train.csv.gz).

Compiled with `cargo --release` and no tricks (i.e. no native flags). `pigz` is version 2.4 installed via ubuntu package manager.

32 threads available for use on benchmark system using NVMe storage.

```bash
cat data.csv | /usr/bin/time crabz -c 3 > crabby.gz
79.34user 4.86system 0:06.52elapsed 1291%CPU (0avgtext+0avgdata 29868maxresident)k
0inputs+3632904outputs (0major+68221minor)pagefaults 0swaps
```

```bash
cat data.csv | /usr/bin/time pigz -3 > crabby.gz
120.65user 12.90system 0:11.36elapsed 1174%CPU (0avgtext+0avgdata 22904maxresident)k
0inputs+3763984outputs (0major+5352minor)pagefaults 0swaps
```

```bash
/usr/bin/time ./target/release/crabz -c 3 -o ./data.csv.gz data.csv
78.56user 3.72system 0:05.99elapsed 1373%CPU (0avgtext+0avgdata 30276maxresident)k
0inputs+3632904outputs (0major+64057minor)pagefaults 0swaps
```

```bash
/usr/bin/time pigz -3 data.csv 
121.20user 9.77system 0:08.01elapsed 1634%CPU (0avgtext+0avgdata 23240maxresident)k
0inputs+3763976outputs (0major+5360minor)pagefaults 0swaps
```

`crabz` seems to be about 20-30% faster than pigz.

This should be apples to apples with the same buffers, number of threads, and compression used.
The round trip md5sums tie out.