# ldeflate
**Pure-Rust parallel DEFLATE/zlib compressor — the write side of [`linflate`].**
`linflate` reads DEFLATE fast. This writes it fast. It exists because serving or repacking
a git packfile means DEFLATE-compressing thousands of objects, and before this crate there
was no compression code anywhere in the znippy-zoomies workspace.
```rust
use ldeflate::{compress_objects, Config};
let objects: Vec<&[u8]> = vec![b"tree 42\0...", b"blob..."];
let streams = compress_objects(&objects, &Config::default())?;
// Each element is a complete RFC-1950 zlib stream. `git` reads them verbatim.
# Ok::<(), ldeflate::DeflateError>(())
```
## What is parallel
Not the entropy coder — that is `zlib-rs`, reused unchanged. The **scheduling** is new, and
it has two axes chosen per object by size, on one no-barrier [`gatling`] pass:
- **small objects are coalesced** into batches, so one task drains many and the expensive
per-stream compressor setup is paid once per batch rather than once per object;
- **large objects are split** at `Z_FULL_FLUSH` boundaries, so a single big blob cannot
serialise a worker.
Git objects are 93.8% under 64 KiB (measured, 48 917 real objects), so they take the first
path. LFS objects are megabytes, so they take the second. **The same `Config` serves both.**
## Guarantees
- **Stock `git` accepts the output.** `tests/stock_git.rs` builds a repository whose entire
object store this crate wrote and runs `git fsck --full --strict` over it, then assembles
a real pack-v2 and runs `git index-pack --strict`. Both have negative controls proving
the checks can fail.
- **Pure Rust.** `flate2` is pinned to `default-features = false, features = ["zlib-rs"]`.
No C, no cmake, no bindgen, no `build.rs`, no build-time network — asserted mechanically
by `tests/no_c_toolchain.rs` against the resolved dependency graph.
- **Deterministic.** Byte-identical output at any worker count: the work plan is a pure
function of the object sizes and the `Config`, and never reads the core count.
- **rayon-free** (constellation LAW 3). All threading routes through `gatling`.
## Measured, oden (Threadripper PRO 3975WX, 32 cores), 2026-08-03
| parallel cost, ~49 k real git objects, ~1.68 GiB, level 6 | **1.025×–1.054× the CPU of serial at 32 threads** — n=7 runs, reproducible to 2.8% |
| efficiency vs cores *actually held* | **94.9%–97.6%** at 32 threads, over occupancies from 10.9 to 24.3 of 32 cores held |
| sustained | 2136 MB/s across 39 consecutive passes, median 100 ms window fully saturated |
| ratio cost of coalescing | **zero** — not one output byte changes |
| ratio cost of splitting | 2.2% at the default 1 MiB segment, only on objects ≥ 4 MiB |
| skipping already-compressed input | **≈20× less CPU** (94.8%–95.6% saved) for **+0.41%** size, on 24 real packfiles |
> **Why no headline “×N faster” figure.** There was one — *28.30× at 32 threads* — and a
> 7-run re-measurement on 2026-08-03 retired it as a *crate* claim: over those runs the
> 32-thread speedup ranged **10.38× to 23.20×** purely with how many cores the box would
> give us, while the CPU cost of the schedule held to 1.025–1.054×. 28.30× was never
> contradicted (it is consistent with, and slightly conservative against, the re-runs), but
> it is unreproducible on a shared machine, so it is not quoted here as a property of the
> code. On a genuinely idle 32-core box the re-measured overhead predicts **30.4×–31.2×**;
> that prediction is untested — oden has not been idle. See Finding 4b in the design doc.
Full tables, the two hypotheses these sweeps falsified, and the honest caveats:
[`../.nornir/ldeflate-design.md`](../.nornir/ldeflate-design.md). The engine patterns it
builds on, written down independently: [`../.nornir/gatling-tricks.md`](../.nornir/gatling-tricks.md).
## When NOT to use it
- **Serving a clone.** The bytes are already deflated in the pack the client pushed;
copying them is a `memcpy` and no compressor beats that.
- **Large already-compressed content** (most LFS objects: video, images, archives). Level-6
DEFLATE measured a compression ratio of **1.000** on high-entropy data — literally
nothing — while running 42× slower than a SHA-1 pass over the same bytes. Route that
content to store-raw *before* it reaches a compressor.
## License
MIT OR Apache-2.0
[`linflate`]: https://crates.io/crates/linflate
[`gatling`]: https://crates.io/crates/rotaryengine