crabka-compression

Kafka wire-protocol compression codecs for Rust.

Part of Crabka, a Rust implementation of Apache Kafka-compatible infrastructure and clients.

Overview

crabka-compression implements the compression formats Apache Kafka uses in record batches. It is a small, I/O-free crate used by crabka-protocol, the producer, and storage code whenever bytes must match Kafka's wire framing.

The crate exposes a stable CompressionType enum that maps to Kafka's record batch attribute bits and compress/decompress helpers for payload bytes.

Codecs

gzip - RFC 1952 gzip streams.
snappy - Kafka's xerial-snappy framing, not Google's Snappy stream format.
lz4 - LZ4 frame format with independent blocks and 64 KiB block size.
zstd - plain zstd frame at level 3.
none - pass-through payloads.

Install

cargo add crabka-compression

For workspace development, use the path dependency from this repository.

Usage

Compress and decompress a Kafka payload:

use crabka_compression::{compress, decompress, CompressionType};

let compressed = compress(CompressionType::Snappy, b"hello kafka")?;
let plain = decompress(CompressionType::Snappy, &compressed, 1024)?;

assert_eq!(plain.as_ref(), b"hello kafka");
# Ok::<(), crabka_compression::CompressionError>(())

decompress requires a caller-provided max_output limit to guard against decompression bombs.

Cargo Features

All codecs are enabled by default. Disable defaults and opt into only the codecs you need:

crabka-compression = { version = "0.3.8", default-features = false, features = ["gzip", "zstd"] }

Calling a disabled codec returns CompressionError::FeatureDisabled.

Documentation

License

Apache-2.0. Derivative work of Apache Kafka; see NOTICE.

crabka-compression 0.3.9