Skip to main content

fv_streams_ops/
lib.rs

1//! # fv-streams-ops — stateful streaming operators
2//!
3//! The line between "fast stateless ETL" and "a streaming engine" is stateful, event-time operators:
4//! windows, keyed state, watermarks, timers, and stream joins. DataFusion 54 has none of these
5//! natively (an aggregate over an unbounded source refuses to plan or runs as batch-in-disguise —
6//! proven by an earlier spike), so we build them.
7//!
8//! ## Event-time windows with keyed incremental aggregation
9//!
10//! - **event time** — each event carries a timestamp; windows are assigned by event time, not arrival.
11//! - **window kinds** — **tumbling** (fixed, non-overlapping `[start, start+size)`) and **sliding**
12//!   (overlapping: an event belongs to every window of width `size` stepping by `slide`, so `ceil(size/
13//!   slide)` windows). Session windows (dynamic bounds + merging) are a separate operator.
14//! - **keyed incremental aggregation** — one running accumulator per `(window, group-key)`, updated
15//!   O(1) per event per window it lands in. (We evaluated reusing `fv_plan::reshape::aggregate` — batch
16//!   GROUP-BY on DataFusion per closed window — but incremental accumulators are the right streaming
17//!   shape: O(1)/event and O(keys × live-windows) memory instead of buffering every row, with no
18//!   per-window query spin-up. The batch-reuse path stays for future *arbitrary-SQL* windowed transforms.)
19//! - **watermark-driven firing** — the watermark is `max-event-time − allowed-lateness`; a window fires
20//!   (emits + evicts) once its end is `≤` the watermark, so late-but-in-lateness events still land and
21//!   genuinely late events (their window already fired) are dropped and counted.
22//!
23//! The core is pure and synchronous — no I/O, no runtime — so it unit-tests in isolation and the stream
24//! worker drives it (feed `push`, drain `advance` after each batch, `flush` on shutdown).
25
26/// Every Rust code block in the README is compiled and run as a doctest.
27#[cfg(doctest)]
28#[doc = include_str!("../README.md")]
29pub struct ReadmeDoctests;
30
31use fv_value::Value;
32
33mod acc;
34pub mod batch_join;
35pub mod batch_window;
36pub mod cells;
37pub mod fields;
38mod frames;
39mod join;
40pub mod join_columns;
41pub mod lookup_join;
42mod rank;
43mod session;
44pub mod sharded_window;
45mod snapshot;
46mod types;
47pub mod values;
48mod watermark;
49mod window_agg;
50mod window_operator;
51pub use batch_join::{FrozenFile, StreamJoinBatch};
52pub use batch_window::{BatchOperator, WindowAggBatch};
53pub use join::{Side, StreamJoin};
54pub use join_columns::{JoinColumns, JoinNaming};
55pub use lookup_join::LookupJoinBatch;
56pub use rank::{LastN, TopN, RANK};
57pub use session::SessionAgg;
58pub use sharded_window::ShardedWindow;
59pub mod error;
60pub use error::{config_fault, delivery_fault, state_fault, transform_fault, StateError};
61pub use types::{Agg, AggOp, Trigger, WindowKind, WindowRow};
62pub use window_agg::WindowAgg;
63pub use window_operator::WindowOperator;