datum-mq

datum-mq is the Kafka satellite crate for Datum. It provides KafkaSource and KafkaSink with at-least-once source offset commits and producer delivery confirmation. Consumers use the native backend by default. The default-on rdkafka Cargo feature keeps both librdkafka compatibility backends selectable; producers now use the native backend by default too. --no-default-features removes rdkafka, rdkafka-sys, and librdkafka's C/CMake build. The native rustls TLS path still compiles aws-lc-rs/aws-lc-sys using CMake, so this is not yet a fully C-toolchain-free build.

The crate itself uses #![forbid(unsafe_code)]; the optional rdkafka backend is an FFI wrapper around librdkafka. Both native paths support plaintext, rustls TLS, SASL PLAIN, and SCRAM-SHA-256/512. Both native paths support Kafka record-batch v2 LZ4 and Zstd compression. Native produce uses Java-compatible murmur2 key partitioning plus sticky round-robin batches for keyless records, and defaults to acks=all plus enable.idempotence=true. Producer id/epoch and per-partition sequences let the broker deduplicate a retried batch, preserving the no-duplicate default that the rdkafka producer provided. Users can explicitly opt out with enable.idempotence=false. Native Produce requests default to the measured winner, one in flight per broker connection; larger bounds are opt-in. Requests for one topic-partition never overlap; after any failure the affected pipeline stops, settles all issued responses, and retries failed partition batches in original order before newer work can issue. Set the bound with KafkaProducerSettings::with_max_in_flight_requests_per_connection; 1 retains the original serial combined-request path. Gzip/Snappy and transactions remain explicit rdkafka boundaries. Unsupported native modes fail fast and name the exact KafkaConsumerBackend::Rdkafka or KafkaProducerBackend::Rdkafka switch; featureless errors also name the missing rdkafka feature. There is no silent fallback.

# Default path: native consumer + producer; both rdkafka backends selectable.
cargo add datum-mq

# Native-only: no librdkafka dependency.
cargo add datum-mq --no-default-features

Transactions and Kafka EOS are deferred. The first stable guarantee is at-least-once: commit consumed offsets only after downstream work has completed, and acknowledge produced records only after Kafka delivery confirmation.