datum-mq

datum-mq is the Kafka satellite crate for Datum. It provides KafkaSource and KafkaSink with at-least-once source offset commits and producer delivery confirmation. Consumers use the native backend by default. The default-on rdkafka Cargo feature keeps the librdkafka consumer selectable and the producer default unchanged; --no-default-features builds only the native consumer and producer backends.

The crate itself uses #![forbid(unsafe_code)]; the optional rdkafka backend is an FFI wrapper around librdkafka. Both native paths support plaintext, rustls TLS, SASL PLAIN, and SCRAM-SHA-256/512. Both native paths support Kafka record-batch v2 LZ4 and Zstd compression. Native produce uses Java-compatible murmur2 key partitioning plus sticky round-robin batches for keyless records, and defaults to acks=all. Its bounded retries are at-least-once but not idempotent: a lost response followed by retry can duplicate records. Native Produce requests default to the measured winner, one in flight per broker connection; larger bounds are opt-in. Requests for one topic-partition never overlap; after any failure the affected pipeline stops, settles all issued responses, and retries failed partition batches in original order before newer work can issue. Set the bound with KafkaProducerSettings::with_max_in_flight_requests_per_connection; 1 retains the original serial combined-request path. Gzip/Snappy and transactions remain explicit rdkafka boundaries. With the rdkafka feature enabled, it stays the default producer; without the feature, the native producer is the only backend and therefore the default. Unsupported native consumer modes fail fast and name KafkaConsumerBackend::Rdkafka; featureless errors also name the missing rdkafka feature. There is no silent fallback.

# Default: native consumer, rdkafka producer, both backends selectable.
cargo add datum-mq

# Native-only: no librdkafka dependency.
cargo add datum-mq --no-default-features

Transactions and Kafka EOS are deferred. The first stable guarantee is at-least-once: commit consumed offsets only after downstream work has completed, and acknowledge produced records only after Kafka delivery confirmation.