spate-kafka 0.1.0

Kafka source and producer sink for the Spate framework, built on rdkafka: a single consumer per process with partition queues fanned across pipeline threads, and a delivery-report-acknowledged producer sink. Applications should depend on the `spate` facade crate with the `kafka` feature.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

//! Kafka producer sink: pipelines terminate in a Kafka topic.
//!
//! The framework owns batching, retry, backpressure, and drain (see
//! `docs/DESIGN.md` § Sink); this connector supplies the two halves of the
//! seam:
//!
//! - **CPU half** ([`KafkaEncoder`] over a [`MessageEncoder`]): on pipeline
//!   threads, each record becomes one Kafka message — key, headers,
//!   payload — appended to the chunk frame in a connector-internal
//!   length-delimited format.
//! - **I/O half** (the sink writer): a shard worker hands the writer a
//!   sealed batch; the writer parses the messages back out, produces them,
//!   and returns only after **every** delivery report confirmed — which is
//!   what lets the framework treat `write_batch → Ok` as the durable-ack
//!   point and advance source watermarks safely.
//!
//! One topic per sink instance: fan-out to multiple topics is the chain's
//! `split` terminal with one Kafka sink per topic. Delivery is
//! **at-least-once, honestly**: `enable.idempotence` (forced on) dedupes
//! only librdkafka's own in-session retries; a framework batch retry after
//! partial delivery re-produces the already-delivered prefix, and crash
//! replay re-produces re-batched data. Nothing here is exactly-once.

mod config;
mod context;
mod encoder;
pub(crate) mod frame;
mod metrics;
mod writer;

pub use config::{Compression, KafkaSink, KafkaSinkConfig, build, from_component_config};
pub use encoder::{
    BytesKeyFn, KafkaBytesEncoder, KafkaEncoder, KafkaJsonEncoder, KafkaMessage, KeyFn,
    MessageEncoder,
};
pub use writer::{KafkaEndpoint, KafkaWriter};