Crate datum_mq 

Kafka sources and sinks for Datum streams.

datum-mq is an additive Datum satellite crate with Tokio-native Kafka consumer and producer backends. The rustls TLS path uses ring (Rust plus assembly), with no AWS-LC C library or CMake build. Native produce and fetch support uncompressed, gzip, Snappy, LZ4, and Zstd record batches. SASL supports PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512; GSSAPI and OAUTHBEARER are not supported.

The consumer’s shipped offset-commit guarantee is at-least-once. KafkaSource::committable emits a datum::SourceWithContext whose context is a KafkaOffset; downstream code commits that offset only after its own checkpoint/side effect succeeds. KafkaSink uses the native producer by default, and its materialized KafkaProducerControl drains final acknowledgements before shutdown. The native producer enables idempotence by default: it negotiates a producer id/epoch and uses per-partition sequences so the broker deduplicates a retried batch. Set enable.idempotence=false to opt out explicitly. The pipeline remains bounded and partition ordered. Kafka transactions and consume-transform-produce EOS are not supported.

Structs

ConsumerRecord

Owned Kafka consumer record.

KafkaBatchOffset

Committable watermark for a batch of Kafka records.

KafkaConfig

Raw Kafka property map with typed helpers for common Datum defaults.

KafkaConsumerSettings

Consumer settings for KafkaSource.

KafkaControl

Materialized control for a Kafka source.

KafkaHeader

Kafka record header.

KafkaMetrics

Cheap cloneable metrics handle for Kafka sources/sinks.

KafkaMetricsSnapshot

Point-in-time metrics snapshot.

KafkaOffset

Context value emitted by KafkaSource::committable.

KafkaOffsetBatch

Batch of offsets for external checkpointing or batched downstream commit sinks.

KafkaPayloadBatch

A payload batch plus one committable Kafka watermark per touched partition.

KafkaPayloadRecord

Payload-only Kafka record emitted by KafkaSource::committable_payload_batches.

KafkaProducerControl

Materialized control for a Kafka producer sink.

KafkaProducerSettings

Producer settings for KafkaSink.

KafkaSink

Kafka producer sink entry points.

KafkaSource

Kafka source entry points.

OffsetCommit

A Kafka commit position. offset is the next offset to consume.

ProducerRecord

Owned Kafka producer record.

TopicPartition

Kafka topic/partition identifier.

TopicPartitionOffset

A concrete topic/partition plus the starting offset for assignment subscriptions.

Enums

CommitPolicy

Offset commit mode for KafkaSource.

KafkaConsumerBackend

Consumer implementation used by KafkaSource.

KafkaProducerBackend

Producer implementation used by crate::KafkaSink.

KafkaTimestamp

Kafka record timestamp.

MqError

Kafka connector errors.

Offset

Kafka assignment start offset.

Subscription

Topic subscription mode.

Constants

VERSION

The datum-mq crate version.

Type Aliases

MqResult

Result type used by datum-mq.