spate_kafka/lib.rs
1//! Kafka source and producer sink for the Spate framework.
2//!
3//! # Source
4//!
5//! Built on `rdkafka` with a **single consumer per process**: partitions
6//! are split into per-partition queues (`split_partition_queue`) and fanned
7//! across pipeline threads as [`SourceLane`](spate_core::source::SourceLane)s,
8//! keeping payload borrows local to the polling thread with zero copies.
9//! Offsets are stored when checkpoint watermarks advance and committed on
10//! an interval; rebalances and shutdown share the framework's drain
11//! choreography, with completion deferred until draining and a synchronous
12//! commit have finished (see [`KafkaSource`] docs).
13//!
14//! One topic per pipeline: the framework's `PartitionId` is the Kafka
15//! partition number. Kafka tombstones (null payloads) surface as empty
16//! payload slices.
17//!
18//! Configuration is read from the pipeline's opaque `source: { kafka: ... }`
19//! section — see [`KafkaSourceConfig`] for the schema and the raw
20//! `rdkafka:` passthrough (framework-owned properties are validated and
21//! rejected with explanations).
22//!
23//! ```yaml
24//! source:
25//! kafka:
26//! brokers: ${KAFKA_BROKERS:-localhost:9092}
27//! topic: orders
28//! group_id: orders-etl
29//! commit_interval: 5s
30//! rdkafka:
31//! fetch.message.max.bytes: "1048576"
32//! ```
33//!
34//! Observability: besides the framework's `spate_source_*` stage metrics, the
35//! source translates librdkafka's statistics snapshot (emitted every
36//! `statistics_interval`, default 5s) into connector-owned
37//! `spate_kafka_source_*` families — transport totals, per-broker health and
38//! round-trip time, client-side queue saturation, and consumer-group
39//! stability. See `docs/METRICS.md` § Kafka source for the full table;
40//! setting `statistics_interval: 0s` disables the families.
41//!
42//! # Sink
43//!
44//! The [`sink`] module is the producer half: pipelines terminate in a
45//! Kafka topic through the framework's sink seam, with a batch
46//! acknowledged only once **every** message's delivery report confirmed —
47//! see the module docs for topology, delivery semantics, and the
48//! `sink: { kafka: ... }` configuration schema.
49//!
50//! This crate deliberately re-exports nothing from `rdkafka`: its types
51//! stay out of public signatures so `rdkafka` major bumps are not breaking
52//! changes here (see `docs/DESIGN.md` § Dependency policy).
53
54mod config;
55mod context;
56mod error;
57mod lane;
58mod metrics;
59mod security;
60pub mod sink;
61mod source;
62
63pub use config::KafkaSourceConfig;
64pub use lane::{KafkaBatch, KafkaLane};
65pub use sink::{
66 KafkaBytesEncoder, KafkaEncoder, KafkaJsonEncoder, KafkaMessage, KafkaSink, KafkaSinkConfig,
67 KafkaWriter, MessageEncoder,
68};
69pub use source::KafkaSource;