datum-mq

datum-mq is the Kafka satellite crate for Datum. It provides KafkaSource and KafkaSink with at-least-once source offset commits and producer delivery confirmation. The default backend is rdkafka/librdkafka; opt-in native consumer and experimental producer backends are available behind the same APIs via KafkaConsumerSettings::with_consumer_backend and KafkaProducerSettings::with_producer_backend.

The crate itself uses #![forbid(unsafe_code)], but the default rdkafka backend is an FFI wrapper around librdkafka. Both native paths support plaintext, rustls TLS, SASL PLAIN, and SCRAM-SHA-256/512. Native produce is uncompressed, uses Java-compatible murmur2 key partitioning plus sticky round-robin batches for keyless records, and defaults to acks=all. Its bounded retries are at-least-once but not idempotent: a lost response followed by retry can duplicate records. Native Produce requests default to the measured winner, one in flight per broker connection; larger bounds are opt-in. Requests for one topic-partition never overlap; after any failure the affected pipeline stops, settles all issued responses, and retries failed partition batches in original order before newer work can issue. Set the bound with KafkaProducerSettings::with_max_in_flight_requests_per_connection; 1 retains the original serial combined-request path. LZ4/Zstd and transactions remain removal/default-flip boundaries; rdkafka stays the default producer.

Transactions and Kafka EOS are deferred. The first stable guarantee is at-least-once: commit consumed offsets only after downstream work has completed, and acknowledge produced records only after Kafka delivery confirmation.