Crate fluxmq 

FluxMQ Core Library

FluxMQ is a high-performance, Kafka-compatible message broker written in Rust. This crate provides the core functionality for the FluxMQ message streaming platform.

Features

• 100% Kafka Compatibility: Full wire protocol compatibility with Apache Kafka
• Ultra-High Performance: 601,379+ messages/second throughput with advanced optimizations
• 20 Kafka APIs: Complete protocol implementation including produce, consume, and admin operations
• Lock-Free Architecture: Lock-free data structures with atomic operations for maximum performance
• Sequential I/O: Log-structured storage with memory-mapped I/O for 20-40x performance gains
• SIMD Optimizations: Hardware-accelerated processing with AVX2/SSE4.2 instructions
• Enterprise Security: TLS/SSL encryption, ACL authorization, SASL authentication
• Consumer Groups: Full coordination with partition assignment and rebalancing
• Distributed Replication: Leader-follower replication with Raft-like consensus

Architecture Overview

FluxMQ implements a modular architecture with the following core components:

• broker - TCP server and request handling
• storage - Hybrid memory-disk storage with crash recovery
• protocol - Kafka wire protocol implementation
• consumer - Consumer group coordination
• replication - Data replication and consensus
• performance - Performance optimization modules
• metrics - Performance monitoring and metrics collection

Quick Start

use fluxmq::{BrokerServer, BrokerConfig};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = BrokerConfig {
        port: 9092,
        host: "0.0.0.0".to_string(),
        enable_consumer_groups: true,
        ..Default::default()
    };
     
    let server = BrokerServer::new(config).await?;
    server.run().await?;
     
    Ok(())
}

Performance Characteristics

FluxMQ achieves exceptional performance through:

• MegaBatch Processing: 1MB batches with LZ4 compression for maximum throughput
• Zero-Copy Operations: Direct memory mapping and bytes::Bytes for efficient I/O
• Lock-Free Metrics: Atomic counters with relaxed memory ordering (3,453% improvement)
• Memory-Mapped Storage: 256MB segments with sequential access patterns
• Hardware Acceleration: SIMD instructions for vectorized message processing

Client Compatibility

FluxMQ is compatible with all major Kafka client libraries:

• Java: org.apache.kafka:kafka-clients v4.1+ (primary target)
• Python: kafka-python library
• Scala: Native Kafka Scala clients
• Go: sarama and confluent-kafka-go
• Node.js: kafkajs and node-rdkafka

Re-exports

pub use broker::BrokerServer;

pub use broker::MessageHandler;

pub use config::BrokerConfig;

pub use consumer::ConsumerGroupConfig;

pub use consumer::ConsumerGroupCoordinator;

pub use consumer::ConsumerGroupManager;

pub use consumer::ConsumerGroupMessage;

pub use http_server::HttpMetricsServer;

pub use metrics::MetricsRegistry;

pub use metrics::MetricsSnapshot;

pub use protocol::FetchRequest;

pub use protocol::FetchResponse;

pub use protocol::Message;

pub use protocol::MetadataRequest;

pub use protocol::MetadataResponse;

pub use protocol::Offset;

pub use protocol::ProduceRequest;

pub use protocol::ProduceResponse;

pub use protocol::Request;

pub use protocol::Response;

pub use replication::BrokerId;

pub use replication::PartitionReplicaInfo;

pub use replication::ReplicationConfig;

pub use replication::ReplicationCoordinator;

pub use replication::ReplicationRole;

pub use storage::HybridStorage;

pub use storage::InMemoryStorage;

Modules

acl

Access Control List (ACL) system for FluxMQ

broker

FluxMQ Broker Module

config

consumer

FluxMQ Consumer Group Module

http_server

HTTP server for metrics and admin endpoints

metrics

FluxMQ Metrics Collection System

performance

FluxMQ Performance Module

protocol

FluxMQ Protocol Module

replication

storage

FluxMQ Storage Module

tls

TLS/SSL support for FluxMQ

topic_manager

FluxMQ Topic Management System

Enums

FluxmqError

FluxMQ error types

Type Aliases

Result

Result type alias for FluxMQ operations