krabka-compression 0.6.0

Kafka wire-protocol compression codecs for Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

# krabka-compression

[![Crates.io](https://img.shields.io/crates/v/krabka-compression.svg)](https://crates.io/crates/krabka-compression)
[![Docs.rs](https://docs.rs/krabka-compression/badge.svg)](https://docs.rs/krabka-compression)
[![CI](https://github.com/krabka-io/krabka-protocol/actions/workflows/ci.yml/badge.svg)](https://github.com/krabka-io/krabka-protocol/actions/workflows/ci.yml)

Kafka wire-protocol compression codecs for Rust.

Part of [Krabka](https://github.com/krabka-io/krabka-protocol), a Rust implementation
of Apache Kafka-compatible infrastructure and clients.

## Overview

`krabka-compression` implements the compression formats Apache Kafka uses in
record batches. It is a small, I/O-free crate. `krabka-protocol`, the producer,
and storage code use it whenever bytes must match Kafka's wire framing.

The crate exposes a stable `CompressionType` enum that maps to Kafka's record
batch attribute bits. It also exposes `compress` and `decompress` helpers for
payload bytes.

## Codecs

- `gzip` - RFC 1952 gzip streams.
- `snappy` - Kafka's xerial-snappy framing, not Google's Snappy stream format.
- `lz4` - LZ4 frame format with independent blocks and 64 KiB block size.
- `zstd` - plain zstd frame at level 3.
- `none` - pass-through payloads.

## Install

```sh
cargo add krabka-compression
```

For workspace development, use the path dependency from this repository.

## Usage

Compress and decompress a Kafka payload:

```rust
use krabka_compression::{compress, decompress, CompressionType};

let compressed = compress(CompressionType::Snappy, b"hello kafka")?;
let plain = decompress(CompressionType::Snappy, &compressed, 1024)?;

assert_eq!(plain.as_ref(), b"hello kafka");
# Ok::<(), krabka_compression::CompressionError>(())
```

`compress_with_level` compresses at a Kafka compression level (KIP-390).
`CompressionType::check_level` checks a level as Kafka's
`CompressionType.levelValidator` does: gzip 1 to 9 or -1, lz4 1 to 17, and zstd
-131072 to 22. The lz4 codec has only the fast compressor, so it writes the
same frame at every level.

`decompress` needs a `max_output` limit from the caller. The limit guards
against decompression bombs.

## Cargo Features

All codecs are enabled by default. Disable the defaults and select only the
codecs you need:

```toml
krabka-compression = { version = "0.6.0", default-features = false, features = ["gzip", "zstd"] }
```

A call to a disabled codec returns `CompressionError::FeatureDisabled`.

## Documentation

- [API documentation](https://docs.rs/krabka-compression)
- [Krabka repository](https://github.com/krabka-io/krabka-protocol)
- [Kafka compatibility matrix](https://github.com/krabka-io/krabka-broker/blob/main/docs/KIP_MATRIX.md)

## License

Apache-2.0. Derivative work of [Apache Kafka](https://kafka.apache.org); see
[NOTICE](https://github.com/krabka-io/krabka-protocol/blob/main/NOTICE).