# libdeflater
> Rust bindings to [libdeflate](https://github.com/ebiggers/libdeflate), a high-performance
> library for working with gzip/zlib/deflate data.
[](https://travis-ci.org/adamkewley/libdeflater)
[](https://crates.io/crates/libdeflater)
[](https://docs.rs/libdeflater)
```
libdeflater = "0.7.2"
```
`libdeflater` is a thin wrapper library around [libdeflate](https://github.com/ebiggers/libdeflate). Libdeflate
is optimal in applications that have the input data up-front, or when (large) input datasets can be split
into smaller chunks (e.g. genomic [bam](https://samtools.github.io/hts-specs/SAMv1.pdf) files, some object stores,
specialized backends, game netcode packets).
This is a thin library around libdeflate that:
- Builds [libdeflate](https://github.com/ebiggers/libdeflate) from source (see
[libdeflate-sys](libdeflate-sys)'s [build.rs](libdeflate-sys/build.rs) file)
- Binds to libdeflate's C API with rust bindings (see [lib.rs](src/lib.rs))
- Contains high-level integration tests to ensure the bindings work (see [integration_test.rs](https://github.com/adamkewley/libdeflater/blob/master/tests/integration_test.rs))
- Contains [usage examples](https://github.com/adamkewley/libdeflater/tree/master/examples) and a
[benchmark suite](https://github.com/adamkewley/libdeflater/tree/master/benches). The benchmark
suite indicates a 2-3x speedup accross the Calgary and Canterbury corpuses (see [Benchmarks](#benchmarks)
section)
⚠️ **Warning**: [libdeflate](https://github.com/ebiggers/libdeflate) is best-suited for *specialized*
use-cases where you know the rough size range of your input up-front. You should use
something like [flate2](https://github.com/alexcrichton/flate2-rs) if you want a
general-purpose deflate library that supports streaming.
# Examples
Example source [here](examples). To run the examples:
```bash
cargo run --example gz_compress.rs
cargo run --example gz_decompress.rs
```
# Benchmarks
Benchmark data is from both the [Calgary Corpus](https://en.wikipedia.org/wiki/Calgary_corpus), and the
[Canterbury Corpus](http://corpus.canterbury.ac.nz/resources/cantrbry.zip). The
benchmark tables below were made with this set of steps:
```bash
wget http://www.data-compression.info/files/corpora/largecalgarycorpus.zip
unzip -d bench_data largecalgarycorpus.zip
wget http://corpus.canterbury.ac.nz/resources/cantrbry.zip
unzip -d bench_data cantrbry.zip
# runs benchmarks against all files in `bench_data`
cargo bench
scripts/process-bench.rb encode
scripts/process-bench.rb decode
```
### Compression
Avg. speedup (on this corpus) is around 2-3x
```
bench size [KB] speedup flate2 [us] libdeflate [us]
alice29.txt 152 3.2 9883 3132
asyoulik.txt 125 3.0 7792 2572
bib 111 2.8 5380 1907
book1 768 3.2 55512 17588
book2 610 2.9 33751 11719
cp.html 24 2.0 571 280
fields.c 11 2.1 229 108
geo 102 8.4 12142 1441
grammar.lsp 3 1.7 64 38
kennedy.xls 1029 6.4 62130 9695
lcet10.txt 426 3.1 23727 7733
news 377 2.7 15814 5898
obj1 21 2.5 543 220
obj2 246 3.7 12202 3336
paper1 53 2.5 2149 865
paper2 82 2.7 4410 1613
paper3 46 2.5 2179 873
paper4 13 2.0 341 174
paper5 11 1.9 269 144
paper6 38 2.3 1350 584
pic 513 2.6 11364 4426
plrabn12.txt 481 3.4 38339 11266
progc 39 2.4 1401 590
progl 71 2.6 2301 897
progp 49 2.5 1322 531
ptt5 513 2.6 11379 4455
sum 38 3.7 1606 435
trans 93 2.5 2403 958
xargs.1 4 1.7 72 42
```
### Decompression
Avg. speedup (on this corpus) is around 2x.
```
bench size [KB] speedup flate2 [us] libdeflate [us]
alice29.txt 152 2.5 601 244
asyoulik.txt 125 2.2 519 234
bib 111 2.5 413 165
book1 768 2.2 3204 1478
book2 610 2.4 2405 987
cp.html 24 2.0 81 41
fields.c 11 2.1 35 17
geo 102 1.8 553 315
grammar.lsp 3 1.8 16 9
kennedy.xls 1029 2.1 2352 1105
lcet10.txt 426 2.5 1505 607
news 377 2.1 1570 738
obj1 21 1.9 94 50
obj2 246 2.3 971 431
paper1 53 2.3 205 90
paper2 82 2.4 341 143
paper3 46 2.1 192 90
paper4 13 1.7 54 32
paper5 11 1.9 52 28
paper6 38 2.1 140 68
pic 513 3.0 1140 386
plrabn12.txt 481 2.2 1973 880
progc 39 2.2 153 70
progl 71 2.5 214 86
progp 49 2.3 135 58
ptt5 513 3.0 1144 386
sum 38 2.1 147 71
trans 93 2.4 246 104
xargs.1 4 1.7 19 11
```
### Benchmark Notes
- All benchmarks are single-threaded
- IO/streaming overhead is not considered. The decompressed data is
read into memory before performing the comparison
- Comparison made against `flate2` with no feature flags (i.e. `miniz`
implementation). `flate2` was chosen because it's the most
popular.
- Comparisons with other `flate2` backends are available on the
`bench-flate2-miniz-oxide` and `bench-flate2-zlib` branches. The
`zlib` backend is ~8 % faster on some of the corpus entries.
- Compression performed with default compression setting in both cases
- Corpus entries were compressed with `flate2` at default compression
level
### Compile-time features
You can enable the following features to customise the build:
- `use_rust_alloc`: Makes libdeflate use Rust's allocator instead of the libc one.
This is useful when Rust is preconfigured to use a
[custom global allocator](https://doc.rust-lang.org/edition-guide/rust-2018/platform-and-target-support/global-allocators.html)
(e.g. pool-based, or a tracking one, or something else entirely).
- `freestanding`: Builds libdeflate in a freestanding mode (no reliance on libc).
This is useful for targets that don't have a C stdlib (e.g. `wasm32-unknown-unknown`)
as otherwise they would fail to compile. Implies `use_rust_alloc`.