Expand description
§fv-streams-ops — stateful streaming operators
The line between “fast stateless ETL” and “a streaming engine” is stateful, event-time operators: windows, keyed state, watermarks, timers, and stream joins. DataFusion 54 has none of these natively (an aggregate over an unbounded source refuses to plan or runs as batch-in-disguise — proven by an earlier spike), so we build them.
§Event-time windows with keyed incremental aggregation
- event time — each event carries a timestamp; windows are assigned by event time, not arrival.
- window kinds — tumbling (fixed, non-overlapping
[start, start+size)) and sliding (overlapping: an event belongs to every window of widthsizestepping byslide, soceil(size/ slide)windows). Session windows (dynamic bounds + merging) are a separate operator. - keyed incremental aggregation — one running accumulator per
(window, group-key), updated O(1) per event per window it lands in. (We evaluated reusingfv_plan::reshape::aggregate— batch GROUP-BY on DataFusion per closed window — but incremental accumulators are the right streaming shape: O(1)/event and O(keys × live-windows) memory instead of buffering every row, with no per-window query spin-up. The batch-reuse path stays for future arbitrary-SQL windowed transforms.) - watermark-driven firing — the watermark is
max-event-time − allowed-lateness; a window fires (emits + evicts) once its end is≤the watermark, so late-but-in-lateness events still land and genuinely late events (their window already fired) are dropped and counted.
The core is pure and synchronous — no I/O, no runtime — so it unit-tests in isolation and the stream
worker drives it (feed push, drain advance after each batch, flush on shutdown).
Re-exports§
pub use batch_join::FrozenFile;pub use batch_join::StreamJoinBatch;pub use batch_window::BatchOperator;pub use batch_window::WindowAggBatch;pub use join_columns::JoinColumns;pub use join_columns::JoinNaming;pub use lookup_join::LookupJoinBatch;pub use sharded_window::ShardedWindow;pub use error::config_fault;pub use error::delivery_fault;pub use error::state_fault;pub use error::transform_fault;pub use error::StateError;
Modules§
- batch_
join - COLUMNAR event-time interval equi-join: the batch-native counterpart of
StreamJoin. - batch_
window - COLUMNAR event-time windowed aggregation: the batch-native counterpart of
WindowAgg. - cells
- A column’s cells in the stable, type-tagged text form that emission keys and group identities
are written in:
nand a number’sf64display,sand a string,band a boolean, a tagged null. The typed accessor is resolved once per column per batch. - error
- The state tier’s errors — what the immutable-file layer under the operators reports when the
filesystem, or a file’s contents, refuse what the runtime asked. Every message names the
operation and the path; where the state is unrecoverable it names the remedy
(
STREAM_RESET_STATE=1), so a FAILED build says what to do, not only what broke. - fields
- AN OUTPUT COLUMN MADE FROM AN INPUT ONE KEEPS THE INPUT’S FIELD: its metadata, such as a
GeoArrow extension name (
geoarrow.point,geoarrow.wkb) or any other Arrow extension, rides along when an operator renames it, widens its type or makes it nullable. An operator never builds an output field from scratch when it has the input’s. - join_
columns - How every join names its output columns — ONE rule, shared by the interval join
(
StreamJoinBatch), the lookup join (LookupJoinBatch) and the row join (StreamJoin), so they can never disagree: - lookup_
join - LOOKUP JOIN — enrich a high-volume STREAM against a BOUNDED side TABLE (Nexmark q13).
- sharded_
window ShardedWindow— aWindowOperatorthat splits a keyBy window intokkey-range shards so the coldest shards can be frozen to files under memory pressure (Phase 3c). Each shard is a full window operator over its own key-range; a row routes tohash(group key) % k, so every key’s rows land in one shard and the shards aggregate independently. Firing is driven by the engine’s watermark (advance_to), which advances every shard’sfired_throughidentically, so lateness and results are identical to one unsharded operator.- values
- A dialect value as Arrow: the type it lands in, a one-row batch of a row, a column of values, and a cell back as a value. The columnar operators’ row seam and the rows the row path still builds use these, so a value lands in Arrow by one set of rules.
Structs§
- Agg
- One named aggregation:
opovercolumn(ignored forCount), output underalias. - LastN
- The last n rows per group key (arrival order — q6’s “last 10 closed auctions”), re-emitting the
standard Agg vocabulary computed over the ring on every update (key
l|<group>worker-side): the group columns, in their input types, then each aggregate. - Session
Agg - A session-window keyed aggregator.
- Stream
Join - An event-time interval equi-join. Feed rows with
push(which returns the joins that row produces against the opposite side); callevictafter each batch to bound state. - TopN
- Top-N rows per group key, ordered by a numeric column (
descending= biggest first — q9/q19), tie broken bytie_byascending (e.g. earliest wins — q9’s dateTime) then by arrival order. Emits, on every change to a key’s set, that key’s CURRENT rows from the first rank that changed down, with a 1-basedrank, as idempotent upserts: the group columns,rank, then the rows’ other columns, in their input types. A row arriving at rank p moves only ranks p and below; the ranks above it keep the row last emitted under their key, so a consumer that keeps the last write per key converges to the same table a full re-emit gives. - Window
Agg - An event-time windowed aggregator. Feed events with
push; after a batch, drain the windows the advancing watermark has closed withadvance;flushfires everything on shutdown. - Window
Row - One fired window’s aggregated output for one group key.
Enums§
- AggOp
- The aggregation op applied to a column within each window+key.
- Side
- Which input stream a row came from.
- Trigger
- When an updating windowed aggregate EARLY-FIRES: emits the current (partial) value of its
still-open windows as upserts, before the watermark closes them. An early emission carries the
same
window_start/window_end/group as the eventual closing fire, so its emission key (w|start|end|group) is identical and downstream last-write-wins converges to the exact closing value. The contract is therefore eventual correctness between fires, exact at close — the closing fire is always the last write for a(window, group). - Window
Kind - How an event time maps to windows.
Constants§
- RANK
- The column a top-N emits each row’s place in, from 1.
Traits§
- Window
Operator - The common interface a windowing operator exposes to the stream worker: feed events with
push, drain the windows the advancing watermark has closed withadvance,flusheverything on shutdown, and register the read partitions so the (min-across-partitions) watermark holds until all report. Implemented byWindowAgg(tumbling/sliding) andSessionAgg(session) so the worker drives either through one path.