Skip to main content

Crate fv_streams_ops

Crate fv_streams_ops 

Source
Expand description

§fv-streams-ops — stateful streaming operators

The line between “fast stateless ETL” and “a streaming engine” is stateful, event-time operators: windows, keyed state, watermarks, timers, and stream joins. DataFusion 54 has none of these natively (an aggregate over an unbounded source refuses to plan or runs as batch-in-disguise — proven by an earlier spike), so we build them.

§Event-time windows with keyed incremental aggregation

  • event time — each event carries a timestamp; windows are assigned by event time, not arrival.
  • window kinds — tumbling (fixed, non-overlapping [start, start+size)) and sliding (overlapping: an event belongs to every window of width size stepping by slide, so ceil(size/ slide) windows). Session windows (dynamic bounds + merging) are a separate operator.
  • keyed incremental aggregation — one running accumulator per (window, group-key), updated O(1) per event per window it lands in. (We evaluated reusing fv_plan::reshape::aggregate — batch GROUP-BY on DataFusion per closed window — but incremental accumulators are the right streaming shape: O(1)/event and O(keys × live-windows) memory instead of buffering every row, with no per-window query spin-up. The batch-reuse path stays for future arbitrary-SQL windowed transforms.)
  • watermark-driven firing — the watermark is max-event-time − allowed-lateness; a window fires (emits + evicts) once its end is ≤ the watermark, so late-but-in-lateness events still land and genuinely late events (their window already fired) are dropped and counted.

The core is pure and synchronous — no I/O, no runtime — so it unit-tests in isolation and the stream worker drives it (feed push, drain advance after each batch, flush on shutdown).

Re-exports§

pub use batch_join::FrozenFile;
pub use batch_join::StreamJoinBatch;
pub use batch_window::BatchOperator;
pub use batch_window::WindowAggBatch;
pub use join_columns::JoinColumns;
pub use join_columns::JoinNaming;
pub use lookup_join::LookupJoinBatch;
pub use sharded_window::ShardedWindow;
pub use error::config_fault;
pub use error::delivery_fault;
pub use error::state_fault;
pub use error::transform_fault;
pub use error::StateError;

Modules§

batch_join
COLUMNAR event-time interval equi-join: the batch-native counterpart of StreamJoin.
batch_window
COLUMNAR event-time windowed aggregation: the batch-native counterpart of WindowAgg.
cells
A column’s cells in the stable, type-tagged text form that emission keys and group identities are written in: n and a number’s f64 display, s and a string, b and a boolean, a tagged null. The typed accessor is resolved once per column per batch.
error
The state tier’s errors — what the immutable-file layer under the operators reports when the filesystem, or a file’s contents, refuse what the runtime asked. Every message names the operation and the path; where the state is unrecoverable it names the remedy (STREAM_RESET_STATE=1), so a FAILED build says what to do, not only what broke.
fields
AN OUTPUT COLUMN MADE FROM AN INPUT ONE KEEPS THE INPUT’S FIELD: its metadata, such as a GeoArrow extension name (geoarrow.point, geoarrow.wkb) or any other Arrow extension, rides along when an operator renames it, widens its type or makes it nullable. An operator never builds an output field from scratch when it has the input’s.
join_columns
How every join names its output columns — ONE rule, shared by the interval join (StreamJoinBatch), the lookup join (LookupJoinBatch) and the row join (StreamJoin), so they can never disagree:
lookup_join
LOOKUP JOIN — enrich a high-volume STREAM against a BOUNDED side TABLE (Nexmark q13).
sharded_window
ShardedWindow — a WindowOperator that splits a keyBy window into k key-range shards so the coldest shards can be frozen to files under memory pressure (Phase 3c). Each shard is a full window operator over its own key-range; a row routes to hash(group key) % k, so every key’s rows land in one shard and the shards aggregate independently. Firing is driven by the engine’s watermark (advance_to), which advances every shard’s fired_through identically, so lateness and results are identical to one unsharded operator.
values
A dialect value as Arrow: the type it lands in, a one-row batch of a row, a column of values, and a cell back as a value. The columnar operators’ row seam and the rows the row path still builds use these, so a value lands in Arrow by one set of rules.

Structs§

Agg
One named aggregation: op over column (ignored for Count), output under alias.
LastN
The last n rows per group key (arrival order — q6’s “last 10 closed auctions”), re-emitting the standard Agg vocabulary computed over the ring on every update (key l|<group> worker-side): the group columns, in their input types, then each aggregate.
SessionAgg
A session-window keyed aggregator.
StreamJoin
An event-time interval equi-join. Feed rows with push (which returns the joins that row produces against the opposite side); call evict after each batch to bound state.
TopN
Top-N rows per group key, ordered by a numeric column (descending = biggest first — q9/q19), tie broken by tie_by ascending (e.g. earliest wins — q9’s dateTime) then by arrival order. Emits, on every change to a key’s set, that key’s CURRENT rows from the first rank that changed down, with a 1-based rank, as idempotent upserts: the group columns, rank, then the rows’ other columns, in their input types. A row arriving at rank p moves only ranks p and below; the ranks above it keep the row last emitted under their key, so a consumer that keeps the last write per key converges to the same table a full re-emit gives.
WindowAgg
An event-time windowed aggregator. Feed events with push; after a batch, drain the windows the advancing watermark has closed with advance; flush fires everything on shutdown.
WindowRow
One fired window’s aggregated output for one group key.

Enums§

AggOp
The aggregation op applied to a column within each window+key.
Side
Which input stream a row came from.
Trigger
When an updating windowed aggregate EARLY-FIRES: emits the current (partial) value of its still-open windows as upserts, before the watermark closes them. An early emission carries the same window_start/window_end/group as the eventual closing fire, so its emission key (w|start|end|group) is identical and downstream last-write-wins converges to the exact closing value. The contract is therefore eventual correctness between fires, exact at close — the closing fire is always the last write for a (window, group).
WindowKind
How an event time maps to windows.

Constants§

RANK
The column a top-N emits each row’s place in, from 1.

Traits§

WindowOperator
The common interface a windowing operator exposes to the stream worker: feed events with push, drain the windows the advancing watermark has closed with advance, flush everything on shutdown, and register the read partitions so the (min-across-partitions) watermark holds until all report. Implemented by WindowAgg (tumbling/sliding) and SessionAgg (session) so the worker drives either through one path.