Crate crab_kafka

crab_kafka

Forward <TCP|UDP> + TLS traffic to kafka.

Based on tokio and rust rdkafka

Basic Usage

It’s strongly encouraged the use of alternative allocator like MiMalloc

use std::collections::HashMap;
use mimalloc::MiMalloc;
use crab_kafka::{forwarder::ForwarderBuilder,Receiver,PartitionStrategies,CheckpointStrategies,TransformStrategies};

#[global_allocator]
static GLOBAL: MiMalloc = MiMalloc;

#[tokio::main]
async fn main() -> Result<(),String> {
    ForwarderBuilder::default()
    .receiver(Receiver::new_tcp_stream("127.0.0.1".to_owned(), "8888".to_owned(), 2000))
    .checkpoint(CheckpointStrategies::OpenDoors)
    .partition(PartitionStrategies::new_sticky_round_robin())
    .transform(TransformStrategies::NoTransform)
    .kafka_settings(HashMap::from([("bootstrap.servers".to_owned(),"broker:29091".to_owned())]))
    .topic("test_topic".to_owned())
    .cache_size(1000)
    .stats_interval(10)
    .build()
    .unwrap()
    .await
}

Examples

• UDP and DTLS example

Modules

• forwarder
  Main library module

Enums

• CheckpointStrategies
  Facilities to control the packets inflow to Kafka
• PartitionStrategies
  Facilities to distributes messages across topic partitions
• Receiver
  Facilities to receive data from sockets
• TransformStrategies
  Facilities to alter the network payload before sending it to Kafka

Traits

• CheckpointStrategy
  Hook to setup per packet forward policy
• PartitionStrategy
  Hook to setup per packet kafka partition policy
• TransformStrategy
  Hook to modify tha packet payload before sending to Kafka