Crate ruststream_rdkafka 

Apache Kafka broker for the RustStream messaging framework, backed by rdkafka / librdkafka.

Transport model

A subscription is one consumer joining one consumer group on one topic; KafkaTopic describes it, and the bare-string #[subscriber("orders")] form consumes the topic named orders through KafkaBroker::default_group. On the publish side OutgoingMessage::name is the destination topic, and a PARTITION_KEY_HEADER header becomes the record’s native key, so Kafka itself keeps per-key ordering.

Settlement follows Kafka’s committed-position model instead of per-message frames; the Commit mode picks how:

• Commit::Auto (the default): librdkafka auto-commit; ack and both nack forms are advisory no-ops (the position is stored when a message is handed to the application, so nack(true) does not cause a redelivery).
• Commit::Tracked: precise at-least-once. ack settles its delivery and the stored position advances across everything settled below it, staying correct under concurrent handler lanes and across offset gaps (transaction markers, compacted topics). nack(false) settles the offset (drop); nack(true) leaves it unsettled, so Kafka redelivers from the committed position on the next fetch of the partition.

Configuration delegates to librdkafka: unset options mean librdkafka defaults, and the raw config(key, value) passthroughs on the broker, the producer, and the descriptor reach every property this crate does not surface as a typed option.

Lazy startup

KafkaBroker::new is synchronous and I/O-free, so a service composes with the synchronous #[ruststream::app] builder; the real network work happens in the idempotent async Broker::connect, called once by the runtime at startup. Publishers handed out before that resolve the shared connection on first use.

Modules

context

Per-delivery context fields exposed to handlers.

testing

In-process test broker, behind the testing feature.

Structs

EosPipeline

An exactly-once pipeline over one transactional producer.

EosReplies

The PublishTransform relaying EOS_SOURCE_HEADER from the originating delivery onto the reply, so the pipeline’s Publisher impl can pair the reply with its consumed offset.

KafkaBroker

An Apache Kafka broker backed by rdkafka / librdkafka.

KafkaMessage

One Kafka delivery: an owned snapshot of the record plus its settlement handle.

KafkaPublisher

A producer handle sharing the broker’s connection.

KafkaSubscriber

A consumer-group member on one topic, yielding KafkaMessage deliveries.

KafkaTopic

A subscription to one Kafka topic through one consumer group.

RoundRobin

A PublishTransform distributing replies round-robin across the first count partitions.

SourceOffset

The source coordinates of one delivery, as EosPipeline::publish needs them.

TransactionalPartitions

Lazily materialized transactional publishers, one per source partition.

Enums

Assignment

The partition assignment strategy for the consumer group (librdkafka’s partition.assignment.strategy).

Commit

How processed deliveries are committed back to the consumer group.

KafkaError

Errors returned by KafkaBroker and the types it hands out.

LaneKey

What drives keyed worker lanes (workers(n, by_key)) for this subscription.

Retry

What nack(true) does on this subscription (see KafkaTopic::retry).

StartOffset

Where a consumer group starts reading when it has no valid committed offset.

Constants

DLQ_SOURCE_OFFSET_HEADER

See DLQ_SOURCE_TOPIC_HEADER.

DLQ_SOURCE_PARTITION_HEADER

See DLQ_SOURCE_TOPIC_HEADER.

DLQ_SOURCE_TOPIC_HEADER

Headers stamped onto a dead-lettered message with the origin of the failed delivery.

EOS_SOURCE_HEADER

Header carrying a transactional delivery’s source coordinates through the reply path.

PARTITION_HEADER

Header naming the explicit destination partition for a publish (an ASCII decimal).

PARTITION_KEY_HEADER

Header carrying a message’s partition key, mapped onto Kafka’s native record key.

RETRY_COUNT_HEADER

Header carrying the number of retry republishes a message has been through.