Crate ruststream_rdkafka

Expand description

Apache Kafka broker for the RustStream messaging framework, backed by rdkafka / librdkafka.

§Transport model

A subscription is one consumer joining one consumer group on one topic; KafkaTopic describes it, and the bare-string #[subscriber("orders")] form consumes the topic named orders through KafkaBroker::default_group. On the publish side OutgoingMessage::name is the destination topic, and a PARTITION_KEY_HEADER header becomes the record’s native key, so Kafka itself keeps per-key ordering.

Settlement follows Kafka’s committed-position model instead of per-message frames; the Commit mode picks how:

Commit::Auto (the default): librdkafka auto-commit; ack and both nack forms are advisory no-ops (the position is stored when a message is handed to the application, so nack(true) does not cause a redelivery).
Commit::Tracked: precise at-least-once. ack settles its delivery and the stored position advances across everything settled below it, staying correct under concurrent handler lanes and across offset gaps (transaction markers, compacted topics). nack(false) settles the offset (drop); nack(true) leaves it unsettled, so Kafka redelivers from the committed position on the next fetch of the partition.

Configuration delegates to librdkafka: unset options mean librdkafka defaults, and the raw config(key, value) passthroughs on the broker, the producer, and the descriptor reach every property this crate does not surface as a typed option.

§The lifecycle ladder

Each state of the connection is its own type, so out-of-order use does not compile:

KafkaBroker::new(servers)      configuration only, synchronous, no I/O
  .connect()   -> ConnectedKafkaBroker   the live connection: subscriptions and publishers
  .shutdown()  -> ClosedKafkaBroker      the terminal witness, carrying the flush result

KafkaBroker::new being synchronous is what lets a service compose with the synchronous #[ruststream::app] builder; the runtime calls connect once at startup.

Publishers follow the same split: KafkaPublish (and its transactional and per-partition transitions) is pure policy, constructible anywhere, with no publish surface at all; pairing it with the connected broker produces the live KafkaPublisher. Handles paired before a shutdown keep aliasing the closed connection, so their operations report KafkaError::Closed rather than succeeding against a dead connection.

One publisher stands outside that split, on purpose: KafkaRetryPublisher, minted from the unconnected broker for builder-time wiring that takes a live publisher rather than a policy - retry_via, the deferred republish behind retry_after that Kafka needs because it has no native delayed redelivery. See its documentation.

Modules§

context: Per-delivery context fields exposed to handlers.
testing: In-process test broker, behind the testing feature.

Structs§

ClosedKafkaBroker: The terminal witness returned by ConnectedBroker::shutdown.
ConnectedKafkaBroker: The connected form of KafkaBroker: the typed witness that Broker::connect succeeded.
EosPipeline: An exactly-once pipeline over one transactional producer, the live form of KafkaEosPublish.
EosReplies: The PublishTransform relaying EOS_SOURCE_HEADER from the originating delivery onto the reply, so the pipeline’s Publisher impl can pair the reply with its consumed offset.
KafkaBroker: An Apache Kafka broker backed by rdkafka / librdkafka.
KafkaEosPublish: The publish policy of EosPipeline: the pipeline id (the producer’s transactional id) plus the commit interval, declared anywhere and paired with the connected broker at startup.
KafkaMessage: One Kafka delivery: an owned snapshot of the record plus its settlement handle.
KafkaPartitionedPublish: The publish policy of TransactionalPartitions, reached from KafkaTransactionalPublish::per_partition.
KafkaPublish: The publish policy of KafkaPublisher: pure declaration, no connection, no publish surface.
KafkaPublisher: A live producer handle on the broker’s shared producer.
KafkaRetryPublisher: A publisher minted from an unconnected KafkaBroker, for the one wiring that cannot take a policy.
KafkaSeeker: Repositions a live KafkaSubscriber, minted by Seekable::seeker.
KafkaSubscriber: A consumer-group member on one topic, yielding KafkaMessage deliveries.
KafkaTopic: A subscription to one Kafka topic through one consumer group.
KafkaTransactionalPublish: The publish policy of KafkaTransactionalPublisher: the transactional mode as its own type, reached from KafkaPublish::transactional_id.
KafkaTransactionalPublisher: A live publisher that produces inside Kafka transactions.
RoundRobin: A PublishTransform distributing replies round-robin across the first count partitions.
SourceOffset: The source coordinates of one delivery, as EosPipeline::publish needs them.
TransactionalPartitions: Transactional publishers, one per source partition, materialized on first use.

Enums§

Assignment: The partition assignment strategy for the consumer group (librdkafka’s partition.assignment.strategy).
Commit: How processed deliveries are committed back to the consumer group.
KafkaError: Errors returned by KafkaBroker and the types it hands out.
KafkaPosition: Where a subscription should resume reading.
LaneKey: What drives keyed worker lanes (workers(n, by_key)) for this subscription.
Retry: What nack(true) does on this subscription (see KafkaTopic::retry).
StartOffset: Where a consumer group starts reading when it has no valid committed offset.

Constants§

DLQ_SOURCE_OFFSET_HEADER: See DLQ_SOURCE_TOPIC_HEADER.
DLQ_SOURCE_PARTITION_HEADER: See DLQ_SOURCE_TOPIC_HEADER.
DLQ_SOURCE_TOPIC_HEADER: Headers stamped onto a dead-lettered message with the origin of the failed delivery.
EOS_SOURCE_HEADER: Header carrying a transactional delivery’s source coordinates through the reply path.
PARTITION_HEADER: Header naming the explicit destination partition for a publish (an ASCII decimal).
PARTITION_KEY_HEADER: Header carrying a message’s partition key, mapped onto Kafka’s native record key.
RETRY_COUNT_HEADER: Header carrying the number of retry republishes a message has been through.