krafka 0.10.0

A pure Rust, async-native Apache Kafka client
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

---
layout: home
title: Home
nav_order: 1
description: "Krafka - A pure Rust, async-native Apache Kafka client"
permalink: /
---

# 🦀 Krafka
{: .fs-9 }

A pure Rust, async-native Apache Kafka client designed for high performance, safety, and ease of use.
{: .fs-6 .fw-300 }

[Get Started]({{ site.baseurl }}/getting-started){: .btn .btn-primary .fs-5 .mb-4 .mb-md-0 .mr-2 }
[View on GitHub](https://github.com/hupe1980/krafka){: .btn .fs-5 .mb-4 .mb-md-0 }

---

## ✨ Features

| Feature | Description |
|:--------|:------------|
| 🦀 **Pure Rust** | No librdkafka or C dependencies |
| ⚡ **Async-native** | Built on Tokio for true async I/O |
| 🔒 **Zero unsafe** | Safe Rust by default |
| 🚀 **High performance** | Zero-copy buffers, inline hot paths, efficient batching |
| 📦 **Full protocol** | Kafka protocol with all compression codecs |
| 🔐 **TLS/SSL** | Using rustls for secure connections |
| 🔑 **SASL** | PLAIN, SCRAM-SHA-256/512 mechanisms |
| 💯 **Transactions** | Exactly-once semantics |
| ☁️ **Cloud-native** | AWS MSK support with IAM auth |
| 🛡️ **Security hardened** | Secret zeroization, constant-time auth, decompression limits |
| 🌐 **SOCKS5 proxy** | Route connections through SOCKS5 proxies (VPN/bastion) |

---

## 🚀 Quick Start

Add Krafka to your `Cargo.toml`:

```toml
[dependencies]
krafka = "0.10.0"
tokio = { version = "1", features = ["full"] }
```

### Producer Example

```rust
use krafka::producer::Producer;

#[tokio::main]
async fn main() -> krafka::error::Result<()> {
    let producer = Producer::builder()
        .bootstrap_servers("localhost:9092")
        .build()
        .await?;

    producer.send("my-topic", Some(b"key"), b"Hello, Kafka!").await?;
    producer.close().await;
    Ok(())
}
```

### Consumer Example

```rust
use krafka::consumer::{Consumer, AutoOffsetReset};
use std::time::Duration;

#[tokio::main]
async fn main() -> krafka::error::Result<()> {
    let consumer = Consumer::builder()
        .bootstrap_servers("localhost:9092")
        .group_id("my-group")
        .auto_offset_reset(AutoOffsetReset::Earliest)
        .build()
        .await?;

    consumer.subscribe(&["my-topic"]).await?;

    loop {
        let records = consumer.poll(Duration::from_secs(1)).await?;
        for record in records {
            println!("{:?}", record);
        }
    }
}
```

---

## 🏗️ Architecture

Krafka is designed with performance and safety as primary goals:

- **Zero-copy buffers** using the `bytes` crate
- **Inline hot paths** for minimal overhead
- **Lazy deserialization** for on-demand record parsing
- **Priority request channels** prevent group ejection during backpressure
- **Multi-connection bundles** for extreme high-throughput scenarios
- **Connection pooling** for efficient broker communication
- **murmur2 hashing** matching Java client for consistent partitioning

---

## 📄 License

Krafka is licensed under the [MIT License](https://github.com/hupe1980/krafka/blob/main/LICENSE).