lbzip2 0.3.0

hmm # lbzip2-rs

🧙‍♂ Med Allfaderns visdom, kompression och korruptionsskydd. ⚡ Med hans blick över varje bit.

Pure Rust parallel bzip2 decompressor. No C dependencies. Usable as a library (in-process, zero-copy) or as a CLI tool:

# Decompress any bzip2 file (including pbzip2 concatenated streams)
cargo run --release --bin lbunzip2 -- planet-241021.osm.bz2 > planet.osm

What makes this crate unique: 100 % Rust (no C/FFI), in-process, zero-copy, and parallel block-boundary scanning — splitting a chunk across N cores is O(N), not O(n) where n is the raw byte count (e.g. 200 MB per chunk). Each core only scans ~500 bytes forward from its split point, so with 16 cores the total scan is ~8 KB for a 200 MB chunk. 4× oversplit lets rayon work-steal across 64 segments, eliminating core idle time from uneven block sizes.

Part of the znippy group of software, designed for fast zero-copy integration with osm-katana — the parallel OSM-to-GeoParquet pipeline.

Performance

Library (in-process, liechtenstein.osm.bz2 — 5.2 MB → 60 MB, 71 blocks)

CLI (lbunzip2 vs C lbzip2)

8-core / 16-thread, NVMe, /dev/null output, 3-run average.

End-to-end: Planet bz2 → PBF (osm-katana)

Full pipeline: bz2 decompress → VTD XML parse → PBF encode, 15 workers.

Usage

use lbzip2::chunk::ChunkDecoder;

let data: &[u8] = /* compressed chunk including BZhN header */;
let decoder = ChunkDecoder::from_header(&data[..4])?;

// Returns segments separately — no giant memcpy
let (segments, consumed) = decoder.decode_chunk_segments(data, true)?;
for seg in &segments {
    // each seg is a Vec<u8> of decompressed data, in order
}

Single-stream sequential API:

let output = lbzip2::stream::decompress(&compressed)?;

License

MIT OR Apache-2.0, plus the original bzip2 license (BSD-style, Julian Seward) for the block-decode routines inspired by the C reference implementation. See LICENSE-BZIP2 for the full text.