Skip to main content

Module compression

Module compression 

Source
Expand description

Record-batch compression codecs.

The Compression enum is always available so crate::record::RecordBatch can read the compression type from attributes regardless of which codec features are enabled. Actual compress/decompress operations require the corresponding Cargo feature:

FeatureCodecBackendPure RustHC mode
gzipGzipflate2yesn/a
snappySnappysnapyesn/a
lz4Lz4lz4_flexyesno
lz4-hcLz4lz4 (C-FFI)noyes
zstdZstdzstd (C-FFI)non/a

compression is a meta-feature that enables gzip + snappy + lz4 + zstd (pure-Rust LZ4). Opt into lz4-hc explicitly if you need High-Compression mode — see the LZ4 section below.

§LZ4 backend selection

  • lz4lz4_flex block API behind a Kafka-compatible custom frame. Fast mode only; the level argument is ignored. Pure Rust, no C toolchain at build time, cross-compile clean.
  • lz4-hclz4 crate (FFI to liblz4) behind the same frame format. Levels 3..=12 use HC mode; levels 0..=2 fall back to fast mode. Requires a C compiler at build time.

Both features are independent — enabling lz4-hc alone is sufficient (the C lib handles fast mode too). When both are enabled, lz4-hc wins at runtime; lz4_flex is linked but unused. Most users should pick one.

For high compression ratio in Kafka workloads, prefer zstd over lz4-hc — modern Kafka clients (Java, librdkafka) default to fast LZ4 when LZ4 is selected, and switch to zstd when ratio matters.

Re-exports§

pub use self::error::CompressionError;
pub use self::error::CompressionErrorKind;

Modules§

error
Error types for crate::compression.
gzip
Gzip codec (flate2 Rust backend).
lz4
Kafka-compatible LZ4 frame codec.
snappy
Snappy codec (snap Rust backend). Level is accepted but ignored.
zstd
Zstandard codec (zstd crate).

Enums§

Compression
Kafka record-batch compression codec, encoded in bits 0–2 of the batch attributes field.

Constants§

MAX_DECOMPRESSED_LEN
Hard ceiling on the decompressed size of a single payload.

Type Aliases§

Result
Result alias for compression operations.