libdeflater 0.1.1

Bindings to libdeflate for DEFLATE (de)compression exposed as non-streaming buffer operations. Contains bindings for raw deflate, zlib, and gzip data.
Documentation

libdeflater

Build Status Crates.io Documentation

Rust bindings to libdeflate. A high-performance library for working with gzip/zlib/deflate data.

libdeflater = "0.1.0"

Warning: libdeflate is for specialized use-cases. You should use something like flate2 if you want a general-purpose deflate library.

libdeflate is optimal in applications that have all input data up and have a mechanism for chunking large input datasets (e.g. genomic bam files, some object stores, specialized backends, game netcode packets). It has a much simpler API than zlib but can't stream data.

Examples

Example source here. To run the examples:

cargo run --example gz_compress.rs
cargo run --example gz_decompress.rs

Benchmarks

Benchmark data is from the Calgary Corpus, which has a decent range of input data types + sizes. See benchmark notes below for more details. The benchmark tables below were made with this set of steps:

wget http://www.data-compression.info/files/corpora/largecalgarycorpus.zip
mkdir bench_data
unzip -d bench_data largecalgarycorpus.zip
cargo bench
scripts/process-bench.rb encode
scripts/process-bench.rb decode

Compression

bench     size [KB]    speedup    flate2 [us]    libdeflate [us]
bib       111          3.2        5310           1678
book1     768          3.4        52818          15691
book2     610          3.1        31973          10447
geo       102          8.7        11447          1316
news      377          2.8        16156          5699
obj1      21           2.7        539            203
obj2      246          3.8        11447          3011
paper1    53           2.7        2206           824
paper2    82           2.9        4298           1501
pic       513          2.8        11438          4082
progc     39           2.6        1426           548
progl     71           2.9        2444           838
progp     49           3.0        1548           524
trans     93           2.8        2432           882

Decompression

bench     size [KB]    speedup    flate2 [us]    libdeflate [us]
bib       111          3.9        662            170
book1     768          3.7        5130           1390
book2     610          4.2        3822           921
geo       102          2.6        764            293
news      377          3.5        2529           714
obj1      21           2.9        136            47
obj2      246          3.7        1485           396
paper1    53           4.0        341            86
paper2    82           3.8        508            134
pic       513          5.7        2248           393
progc     39           3.6        244            68
progl     71           4.5        376            84
progp     49           4.3        262            61
trans     93           4.4        448            101

Benchmark Notes

  • Benchmark data is from the Calgary Corpus, which has a decent range of input data types + sizes.

  • Benchmarks were ran by unpacking the corpus into bench_data in this repo then running cargo bench which, in turn, runs this custom benchmark suite configured with this config.

  • The results were aggregated with this ruby script that extracts Mean.point_estimate from Criterion's estimates.json

  • All benchmarks are single-threaded

  • Comparison made against flate2 with no feature flags (i.e. miniz implementation). flate2 was chosen because it's the most popular.

  • Compression performed with default compression setting in both cases

  • Corpus entries were compressed with flate2 at default compression level