fv-streams-ops
The stateful operators behind FusionVault Streams: windows, sessions, interval joins and rankings.
They are pure and synchronous, with no I/O and no runtime. A host pushes events in, collects the results that fire, and asks for a snapshot when it takes a checkpoint.
A tumbling window
use ;
use Value;
// 10-second tumbling windows, a 2-second watermark delay, count + sum per `auction`.
let mut w = tumbling;
w.expect_partitions;
let row = ;
w.push; // partition 0, event time 1s → the [0, 10s) window
w.push;
// Nothing fires until the watermark (max event time − delay) passes a window's end:
// after 4s the watermark is 2s, so the window is still open.
assert!;
w.push; // 13s → watermark 11s ≥ 10s: the first window can fire
let fired = w.advance;
assert_eq!;
assert_eq!;
assert_eq!;
assert_eq!;
pushreturnsfalsefor an event that arrives after every window it belongs to has fired.late_dropped()counts those events.flush()fires every window that is still open, at the end of the input. An ordered stop does not call it: open windows stay in the checkpoint.
The operators
| Operator | What it does |
|---|---|
WindowAgg |
Tumbling or sliding windows on event time. It keeps one running accumulator per window and group key, so each event costs O(1). A window fires when the watermark passes its end. |
SessionAgg |
Session windows. A gap with no events closes a session, and overlapping sessions merge. |
StreamJoin |
Joins two inputs on a key, pairing events within a time window. Each side's buffer is bounded, and the watermark evicts old rows. |
TopN |
Keeps the top n rows per key, by a column, and emits changes as upserts. |
LastN |
Keeps the last n aggregated rows per key, in a ring, and emits changes as upserts. |
What every operator shares:
- It implements
WindowOperator, withpush,advanceandflush. - It keeps a watermark per partition, with an idle timeout (
with_idle_timeout), so a quiet partition cannot hold results back. - It saves and restores its state with
snapshot()andload_snapshot(). The encoding is versioned, and carries a fingerprint of the operator's configuration. This is what lets the engine checkpoint and restore an operator exactly.
Aggregations
| Kind | AggOp |
|---|---|
| Basic | Count, Sum, Avg, Min, Max and CountDistinct |
| Spread, computed online with Welford's method | VarPop, VarSamp, StddevPop and StddevSamp |
| Boolean | BoolAnd and BoolOr |
| Bitwise | BitAnd, BitOr and BitXor |
Build one with Agg::count("alias") or Agg::of(op, "column", "alias").
A pipeline names them as the
aggregations reference
shows.
Licence
Apache 2.0.