Crate spate_kafka 

Kafka source and producer sink for the Spate framework.

Source

Built on rdkafka with a single consumer per process: partitions are split into per-partition queues (split_partition_queue) and fanned across pipeline threads as SourceLanes, keeping payload borrows local to the polling thread with zero copies. Offsets are stored when checkpoint watermarks advance and committed on an interval; rebalances and shutdown share the framework’s drain choreography, with completion deferred until draining and a synchronous commit have finished (see KafkaSource docs).

One topic per pipeline: the framework’s PartitionId is the Kafka partition number. Kafka tombstones (null payloads) surface as empty payload slices.

Configuration is read from the pipeline’s opaque source: { kafka: ... } section — see KafkaSourceConfig for the schema and the raw rdkafka: passthrough (framework-owned properties are validated and rejected with explanations).

source:
  kafka:
    brokers: ${KAFKA_BROKERS:-localhost:9092}
    topic: orders
    group_id: orders-etl
    commit_interval: 5s
    rdkafka:
      fetch.message.max.bytes: "1048576"

Observability: besides the framework’s spate_source_* stage metrics, the source translates librdkafka’s statistics snapshot (emitted every statistics_interval, default 5s) into connector-owned spate_kafka_source_* families — transport totals, per-broker health and round-trip time, client-side queue saturation, and consumer-group stability. See docs/METRICS.md § Kafka source for the full table; setting statistics_interval: 0s disables the families.

Sink

The sink module is the producer half: pipelines terminate in a Kafka topic through the framework’s sink seam, with a batch acknowledged only once every message’s delivery report confirmed — see the module docs for topology, delivery semantics, and the sink: { kafka: ... } configuration schema.

This crate deliberately re-exports nothing from rdkafka: its types stay out of public signatures so rdkafka major bumps are not breaking changes here (see docs/DESIGN.md § Dependency policy).

Re-exports

pub use sink::KafkaBytesEncoder;

pub use sink::KafkaEncoder;

pub use sink::KafkaJsonEncoder;

pub use sink::KafkaMessage;

pub use sink::KafkaSink;

pub use sink::KafkaSinkConfig;

pub use sink::KafkaWriter;

pub use sink::MessageEncoder;

Modules

sink

Kafka producer sink: pipelines terminate in a Kafka topic.

Structs

KafkaBatch

One poll’s worth of messages, borrowed from the lane.

KafkaLane

One assigned partition’s pollable queue.

KafkaSource

Kafka source: one consumer-group member per process, partitions split into per-lane queues polled by pipeline threads. Constructed from config (KafkaSource::new) or a pipeline component section (KafkaSource::from_component_config).

KafkaSourceConfig

Configuration of a KafkaSource, deserialized from the pipeline’s opaque source: { kafka: ... } section.