libdeflater
Rust bindings to libdeflate. A high-performance library for working with gzip/zlib/deflate data.
libdeflater = "0.1.0"
Warning: libdeflate is for specialized use-cases. You should use something like flate2 if you want a general-purpose deflate library.
libdeflate is optimal in applications that have all input data up and have a mechanism for chunking large input datasets (e.g. genomic bam files, some object stores, specialized backends, game netcode packets). It has a much simpler API than zlib but can't stream data.
Examples
Example source here. To run the examples:
Benchmarks
Benchmark data is from the Calgary Corpus, which has a decent range of input data types + sizes. See benchmark notes below for more details. The benchmark tables below were made with this set of steps:
Compression
bench size [KB] speedup flate2 [us] libdeflate [us]
bib 111 3.2 5310 1678
book1 768 3.4 52818 15691
book2 610 3.1 31973 10447
geo 102 8.7 11447 1316
news 377 2.8 16156 5699
obj1 21 2.7 539 203
obj2 246 3.8 11447 3011
paper1 53 2.7 2206 824
paper2 82 2.9 4298 1501
pic 513 2.8 11438 4082
progc 39 2.6 1426 548
progl 71 2.9 2444 838
progp 49 3.0 1548 524
trans 93 2.8 2432 882
Decompression
bench size [KB] speedup flate2 [us] libdeflate [us]
bib 111 3.9 662 170
book1 768 3.7 5130 1390
book2 610 4.2 3822 921
geo 102 2.6 764 293
news 377 3.5 2529 714
obj1 21 2.9 136 47
obj2 246 3.7 1485 396
paper1 53 4.0 341 86
paper2 82 3.8 508 134
pic 513 5.7 2248 393
progc 39 3.6 244 68
progl 71 4.5 376 84
progp 49 4.3 262 61
trans 93 4.4 448 101
Benchmark Notes
-
Benchmark data is from the Calgary Corpus, which has a decent range of input data types + sizes.
-
Benchmarks were ran by unpacking the corpus into
bench_data
in this repo then runningcargo bench
which, in turn, runs this custom benchmark suite configured with this config. -
The results were aggregated with this ruby script that extracts
Mean.point_estimate
fromCriterion
'sestimates.json
-
All benchmarks are single-threaded
-
Comparison made against
flate2
with no feature flags (i.e.miniz
implementation).flate2
was chosen because it's the most popular. -
Compression performed with default compression setting in both cases
-
Corpus entries were compressed with
flate2
at default compression level