spate

The front door to the Spate framework: a high-performance, at-least-once ETL pipeline library for Rust. You write the operator graph in Rust and it chains into a single monomorphized loop over zero-copy borrowed records on pinned processing threads, with checkpoint-driven source commits, sharded asynchronous sinks, backpressure, and first-class Prometheus metrics.

[dependencies]
spate = { version = "0.1", features = ["full"] }

This crate re-exports the engine and every connector behind a feature flag, so an application depends on one crate rather than eight. Nothing is on by default — a pipeline that only writes to ClickHouse never compiles the Kafka tree, and never resolves rdkafka into its lockfile.

Start with examples/memory_pipeline.rs, which runs without any infrastructure. examples/kafka_avro_to_clickhouse.rs is the flagship assembly and ships with its YAML alongside it.

Delivery semantics

At-least-once, and the framework is built around not quietly weakening that: a source watermark is never committed past unacknowledged data, including across rebalances and shutdown. Records that fail are either skipped or fail the pipeline — there is no third policy that silently drops them — and both outcomes are counted in metrics rather than only logged.

Duplicates are therefore possible after a crash. Sinks that need effectively once should be idempotent or deduplicating; the ClickHouse guide covers how to get there with ReplacingMergeTree and with insert deduplication.

Full documentation is at spate.kainth.dev.