datum-mq

datum-mq is Datum's native Kafka satellite crate. It provides KafkaSource and KafkaSink with at-least-once source offset commits and producer delivery confirmation. The default feature set is empty; consumer and producer traffic always uses the Tokio-native client.

The crate uses #![forbid(unsafe_code)]. Its rustls TLS path uses ring (Rust plus assembly), so it does not compile AWS-LC or a CMake-built TLS library. Zstd codec support still uses zstd-sys; removing the Kafka FFI client does not make the crate fully C-free.

The client supports plaintext, rustls TLS, SASL PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. It handles uncompressed, gzip, Snappy, LZ4, and Zstd Kafka record-batch v2 payloads, so the native-only client loses no compression codec. SASL GSSAPI/OAUTHBEARER and Kafka transactions/consume-transform-produce EOS are not supported.

Native production uses Java-compatible murmur2 key partitioning plus sticky round-robin batches for keyless records, and defaults to acks=all plus enable.idempotence=true. Producer id/epoch and per-partition sequences let the broker deduplicate retried batches. Users can explicitly opt out with enable.idempotence=false. Native Produce requests default to one in flight per broker connection; larger bounds are opt-in through KafkaProducerSettings::with_max_in_flight_requests_per_connection. Requests for one topic-partition never overlap, and retry recovery preserves partition order.

cargo add datum-mq

The source guarantee is at-least-once: commit consumed offsets only after downstream work has completed, and acknowledge produced records only after Kafka delivery confirmation.