znippy-common 0.9.0

Core logic and data structures for Znippy, a parallel chunked compression system.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

![znippy](https://github.com/user-attachments/assets/7db1c1c1-d577-4f87-bfe1-11af6e8c58a0)

# znippy

Arrow IPC archive format with parallel compress/decompress, package-aware metadata extraction, and random-access reads. Natively readable by DuckDB, Polars, and DataFusion without a znippy library.

**6,310 MB/s decompress · 5,802 MB/s compress · 32 cores · v0.7.2**

| Test | Compress | Decompress | Files |
|---|---|---|---|
| mixed repo 530 MB | 3,118 MB/s | 6,310 MB/s | 6 |
| single file 2 GB | 5,802 MB/s | 3,066 MB/s | 1 |
| 100k small files 10 KB | 4,811 MB/s | 1,957 MB/s | 100,000 |
| random (incompressible) | 105 MB/s | 2,890 MB/s | 1 |
| real jars RAID NVMe | 2,527 MB/s | **9,950 MB/s** | 4,730 |

## Benchmark history

| Version | Date | Best decompress | Machine |
|---|---|---|---|
| 0.7.2 | 2026-05-24 | 9,950 MB/s (real jars) | 32-core |