spate-core 0.1.0

Engine for the Spate framework: records, operator chains, source/sink abstractions, checkpointing, backpressure, config, metrics, and the pipeline runtime. Applications should depend on the `spate` facade crate instead.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

//! Checkpointing: acknowledgement tracking and watermark commits.
//!
//! One [`AckRef`] is created per source poll batch (by an [`AckIssuer`] on
//! the polling thread) and cloned into every record derived from that batch
//! — filter drops, `flat_map` fan-out, and multi-sink routing all compose
//! through plain `Clone`/`Drop`. When the last clone drops, the batch
//! resolves with the worst status observed and a message flows to the
//! [`Checkpointer`], which advances per-partition contiguous watermarks
//! ([`PartitionTracker`]) and hands `Source::commit` its positions on an
//! interval.
//!
//! Invariants (see `docs/DESIGN.md` and `CLAUDE.md`):
//! - the tracker stays synchronous and tokio-free (loom-tested);
//! - the ack path never blocks (unbounded channels, atomics only);
//! - a watermark never advances past an unacknowledged or failed batch,
//!   including across rebalances (assignment epochs make stale
//!   acknowledgements harmless).
//!
//! The concurrency-bearing primitives are model-checked with
//! [loom](https://docs.rs/loom):
//!
//! ```text
//! RUSTFLAGS="--cfg loom" cargo test -p spate-core --release checkpoint::loom_tests
//! ```

mod ack;
#[cfg(not(loom))]
mod checkpointer;
#[cfg(all(test, loom))]
mod loom_tests;
mod sync;
mod tracker;

pub use ack::{AckMsg, AckRef, AckSet, AckStatus, BatchId};
#[cfg(not(loom))]
pub use checkpointer::{AckIssuer, Checkpointer, DrainStats};
pub use tracker::{PartitionTracker, ResolveOutcome};