krabka-compression

Kafka wire-protocol compression codecs for Rust.

Part of Krabka, a Rust implementation of Apache Kafka-compatible infrastructure and clients.

Overview

krabka-compression implements the compression formats Apache Kafka uses in record batches. It is a small, I/O-free crate. krabka-protocol, the producer, and storage code use it whenever bytes must match Kafka's wire framing.

The crate exposes a stable CompressionType enum that maps to Kafka's record batch attribute bits. It also exposes compress and decompress helpers for payload bytes.

Codecs

gzip - RFC 1952 gzip streams.
snappy - Kafka's xerial-snappy framing, not Google's Snappy stream format.
lz4 - LZ4 frame format with independent blocks and 64 KiB block size.
zstd - plain zstd frame at level 3.
none - pass-through payloads.

Install

cargo add krabka-compression

For workspace development, use the path dependency from this repository.

Usage

Compress and decompress a Kafka payload:

use krabka_compression::{compress, decompress, CompressionType};

let compressed = compress(CompressionType::Snappy, b"hello kafka")?;
let plain = decompress(CompressionType::Snappy, &compressed, 1024)?;

assert_eq!(plain.as_ref(), b"hello kafka");
# Ok::<(), krabka_compression::CompressionError>(())

compress_with_level compresses at a Kafka compression level (KIP-390). CompressionType::check_level checks a level as Kafka's CompressionType.levelValidator does: gzip 1 to 9 or -1, lz4 1 to 17, and zstd -131072 to 22. The lz4 codec has only the fast compressor, so it writes the same frame at every level.

decompress needs a max_output limit from the caller. The limit guards against decompression bombs.

Cargo Features

All codecs are enabled by default. Disable the defaults and select only the codecs you need:

krabka-compression = { version = "0.5.1", default-features = false, features = ["gzip", "zstd"] }

A call to a disabled codec returns CompressionError::FeatureDisabled.

Documentation

License

Apache-2.0. Derivative work of Apache Kafka; see NOTICE.

krabka-compression 0.5.1