rust-with-kafka-tls 1.0.1

Rust messaging with a Strimzi Kafka cluster secured with self-signed tls assets for encryption in transit with mTLS for client authentication
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

# Rust Kafka Publisher and Subscriber Demo with Strimzi Kafka and Client mTLS for encryption in transit

Simple producer/consumer messaging example that works with a Kafka cluster that enforces client mTLS authentication. 

## Optional - Custom TLS Assets

By default the ``./kubernetes/deploy.sh`` script will use the included tls assets in the repo: [./kubernetes/tls](https://github.com/jay-johnson/rust-with-strimzi-kafka-and-tls/tree/main/kubernetes/tls). Before going into production with these, please change these to your own to prevent security issues.

If you want to use your own tls assets you can set these environment variables:

- ``CA_FILE`` - path to your Certificate Authority (CA) file
- ``CA_KEY_FILE`` - path to your CA key file
- ``TLS_CHAIN_FILE`` - path to your tls server chain file (ordered by: cert then CA)
- ``TLS_KEY_FILE`` - path to your tls server key file

```bash
./kubernetes/deploy.sh
```

## Verify Client mTLS

Clients must provide the tls key, cert and CAfile for establishing a valid mutual tls connection.

For local testing you will need to add these entries to your ``/etc/hosts`` or set up a real nameserver for dns:

- ``cluster-0-broker-0.redten.io``
- ``cluster-0-broker-1.redten.io``
- ``cluster-0-broker-2.redten.io``

As an example on the local loopback device:

```bash
# /etc/hosts
127.0.0.1      cluster-0-broker-0.redten.io cluster-0-broker-1.redten.io cluster-0-broker-2.redten.io
```

For users on minikube you can use ``minikube ip -p CLUSTERNAME`` to get the ip address:

```bash
# /etc/hosts
192.168.49.2   cluster-0-broker-0.redten.io cluster-0-broker-1.redten.io cluster-0-broker-2.redten.io
```

```bash
echo "ssl test" | openssl s_client -connect \
    cluster-0-broker-0.redten.io:32151 \
    -key ./kubernetes/tls/client-key.pem \
    -cert ./kubernetes/tls/client.pem \
    -CAfile ./kubernetes/tls/ca.pem \
    -verify_return_error \
    && echo "strimzi kafka cluster is working with self-signed tls assets!"
```

## Create Kafka Topic for Rust Messaging

```bash
cat <<EOL | kubectl apply -n dev -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: testing
  labels:
    strimzi.io/cluster: "dev"
spec:
  partitions: 3
  replicas: 3
EOL
```

## Rust Messaging

### Set TLS Paths

You can either copy the TLS assets into the ``./tls`` directory or export the environment variables:

- ``KAFKA_TLS_CLIENT_CA`` - path to the Certificate Authority file
- ``KAFKA_TLS_CLIENT_KEY`` - path to the server key file
- ``KAFKA_TLS_CLIENT_CERT`` - path to the server certificate file

### Set Broker Addresses

Export this environment variable to the correct broker fqdns and ports:

- ``KAFKA_BROKERS`` - comma delimited list of kafka brokers (format: ``cluster-0-broker-0.redten.io:32151,cluster-0-broker-1.redten.io:32152,cluster-0-broker-2.redten.io:32153``)

### Start Consumer

```bash
# export KAFKA_BROKERS=cluster-0-broker-0.redten.io:32151,cluster-0-broker-1.redten.io:32152,cluster-0-broker-2.redten.io:32153
cargo build --example run-consumer
export RUST_BACKTRACE=1
export RUST_LOG=info
./target/debug/examples/run-consumer -b $KAFKA_BROKERS -g rust-consumer-testing -t testing
```

### Start Producer

```bash
# export KAFKA_BROKERS=cluster-0-broker-0.redten.io:32151,cluster-0-broker-1.redten.io:32152,cluster-0-broker-2.redten.io:32153
cargo build --example run-producer
export RUST_BACKTRACE=1
export RUST_LOG=info
./target/debug/examples/run-producer -b $KAFKA_BROKERS -t testing
```

## Sources

- Rust Consumer and Producer examples from [rdkafka](https://github.com/fede1024/rust-rdkafka) with examples: https://github.com/fede1024/rust-rdkafka/tree/master/examples
- Using your own CA and TLS Assets with Strimzi: https://github.com/scholzj/strimzi-custom-ca-test
- This crate was created to profile Strimzi kafka cluster performance from this [blog article](https://jaypjohnson.com/2022-09-14-deploying-kafka-clusters-with-tls-on-kubernetes-using-strimzi-and-helm.html)