Crate crab_kafka

crab_kafka

Forward <TCP|UDP> + TLS traffic to kafka.

Based on tokio and rust rdkafka

Basic Usage

It’s strongly encouraged the use of alternative allocator like MiMalloc

use anyhow::Result;
use std::collections::HashMap;
use mimalloc::MiMalloc;
use crab_kafka::{forwarder::ForwarderBuilder,Receiver,PartitionStrategies,CheckpointStrategies,TransformStrategies};

#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;

#[tokio::main]
async fn main() -> Result<()> {
    ForwarderBuilder::default()
    .receiver(Receiver::new_tcp_stream("127.0.0.1".to_owned(), "8888".to_owned(), 2000))
    .checkpoint(CheckpointStrategies::OpenDoors)
    .partition(PartitionStrategies::new_sticky_round_robin())
    .transform(TransformStrategies::NoTransform)
    .kafka_settings(HashMap::from([("bootstrap.servers".to_owned(),"broker:29091".to_owned())]))
    .topic("test_topic".to_owned())
    .cache_size(1000)
    .stats_interval(10)
    .build()
    .unwrap()
    .await
}

Examples

• UDP and DTLS example

Modules

forwarder

Main library module

Enums

CheckpointStrategies

Facilities to control the packets inflow to Kafka

PartitionStrategies

Facilities to distributes messages across topic partitions

Receiver

Facilities to receive data from sockets

TransformStrategies

Facilities to alter the network payload before sending it to Kafka

Traits

CheckpointStrategy

Hook to setup per packet forward policy

PartitionStrategy

Hook to setup per packet kafka partition policy

TransformStrategy

Hook to modify tha packet payload before sending to Kafka