fv-streams-ops 0.9.0

Stateful streaming operators for FusionVault Streams: event-time tumbling and sliding windows with keyed incremental aggregation, session windows, stream joins, top-N and last-N, watermarks and snapshots. Pure and synchronous — no I/O, no runtime.
Documentation

fv-streams-ops

The stateful operators behind FusionVault Streams: windows, sessions, interval joins and rankings.

They are pure and synchronous, with no I/O and no runtime. A host pushes events in, collects the results that fire, and asks for a snapshot when it takes a checkpoint.

A tumbling window

use fv_streams_ops::{Agg, AggOp, WindowAgg};
use fv_value::Value;

// 10-second tumbling windows, a 2-second watermark delay, count + sum per `auction`.
let mut w = WindowAgg::tumbling(10_000, 2_000, vec!["auction".into()],
    vec![Agg::count("bids"), Agg::of(AggOp::Sum, "price", "total")]);
w.expect_partitions(&[0], 0);

let row = |auction: f64, price: f64| vec![
    ("auction".to_string(), Value::Num(auction)),
    ("price".to_string(), Value::Num(price)),
];
w.push(0, 1_000, &row(1.0, 10.0));    // partition 0, event time 1s → the [0, 10s) window
w.push(0, 4_000, &row(1.0, 15.0));

// Nothing fires until the watermark (max event time − delay) passes a window's end:
// after 4s the watermark is 2s, so the window is still open.
assert!(w.advance(4_000).is_empty());

w.push(0, 13_000, &row(1.0, 7.0));    // 13s → watermark 11s ≥ 10s: the first window can fire
let fired = w.advance(13_000);
assert_eq!(fired.len(), 1);
assert_eq!(fired[0].key, vec![Value::Num(1.0)]);
assert_eq!((fired[0].window_start, fired[0].window_end), (0, 10_000));
assert_eq!(fired[0].aggs, vec![("bids".to_string(), Value::Num(2.0)), ("total".to_string(), Value::Num(25.0))]);
  • push returns false for an event that arrives after every window it belongs to has fired. late_dropped() counts those events.
  • flush() fires every window that is still open, at the end of the input. An ordered stop does not call it: open windows stay in the checkpoint.

The operators

Operator What it does
WindowAgg Tumbling or sliding windows on event time. It keeps one running accumulator per window and group key, so each event costs O(1). A window fires when the watermark passes its end.
SessionAgg Session windows. A gap with no events closes a session, and overlapping sessions merge.
StreamJoin Joins two inputs on a key, pairing events within a time window. Each side's buffer is bounded, and the watermark evicts old rows.
TopN Keeps the top n rows per key, by a column, and emits changes as upserts.
LastN Keeps the last n aggregated rows per key, in a ring, and emits changes as upserts.

What every operator shares:

  • It implements WindowOperator, with push, advance and flush.
  • It keeps a watermark per partition, with an idle timeout (with_idle_timeout), so a quiet partition cannot hold results back.
  • It saves and restores its state with snapshot() and load_snapshot(). The encoding is versioned, and carries a fingerprint of the operator's configuration. This is what lets the engine checkpoint and restore an operator exactly.

Aggregations

Kind AggOp
Basic Count, Sum, Avg, Min, Max and CountDistinct
Spread, computed online with Welford's method VarPop, VarSamp, StddevPop and StddevSamp
Boolean BoolAnd and BoolOr
Bitwise BitAnd, BitOr and BitXor

Build one with Agg::count("alias") or Agg::of(op, "column", "alias"). A pipeline names them as the aggregations reference shows.

Licence

Apache 2.0.