fv-streams-ops 0.6.0

Stateful streaming operators for FusionVault Streams: event-time tumbling and sliding windows with keyed incremental aggregation, session windows, stream joins, top-N and last-N, watermarks and snapshots. Pure and synchronous — no I/O, no runtime.
Documentation

fv-streams-ops

The stateful operators behind FusionVault Streams: event-time windows with keyed incremental aggregation, session windows, stream-stream interval joins, and top-N / last-N ranking. Pure and synchronous: no I/O, no runtime, no Kafka. A host feeds events in, drains fired results out, and asks for a snapshot when it checkpoints.

A tumbling window

use fv_streams_ops::{Agg, AggOp, WindowAgg};
use fv_value::Value;

// 10-second tumbling windows, 2 seconds of allowed lateness, count + sum per `auction`.
let mut w = WindowAgg::tumbling(10_000, 2_000, vec!["auction".into()],
    vec![Agg::count("bids"), Agg::of(AggOp::Sum, "price", "total")]);
w.expect_partitions(&[0], 0);

let row = |auction: f64, price: f64| vec![
    ("auction".to_string(), Value::Num(auction)),
    ("price".to_string(), Value::Num(price)),
];
w.push(0, 1_000, &row(1.0, 10.0));    // partition 0, event time 1s → the [0, 10s) window
w.push(0, 4_000, &row(1.0, 15.0));

// Nothing fires until the watermark (max event time − lateness) passes a window's end:
// after 4s the watermark is 2s, so the window is still open.
assert!(w.advance(4_000).is_empty());

w.push(0, 13_000, &row(1.0, 7.0));    // 13s → watermark 11s ≥ 10s: the first window can fire
let fired = w.advance(13_000);
assert_eq!(fired.len(), 1);
assert_eq!(fired[0].key, vec![Value::Num(1.0)]);
assert_eq!((fired[0].window_start, fired[0].window_end), (0, 10_000));
assert_eq!(fired[0].aggs, vec![("bids".to_string(), Value::Num(2.0)), ("total".to_string(), Value::Num(25.0))]);

push returns false for an event that is late beyond the lateness bound (its window already fired); late_dropped() counts them. flush() fires everything still open at shutdown.

The operators

  • WindowAgg — tumbling or sliding windows over event time; one running accumulator per (window, group key), O(1) per event; fires when the watermark passes a window's end.
  • SessionAgg — session windows: a gap closes a session, overlapping sessions merge.
  • StreamJoin — a two-input interval join on a key within a time window, with a bounded buffer on each side and eviction by watermark.
  • TopN / LastN — unbounded per-key rankings: the top n rows by a column, or a ring of the last n aggregated rows; emit as upserts.

All of them implement WindowOperator (push / advance / flush), carry a per-partition watermark with a bounded-idleness timeout (with_idle_timeout) so a quiet partition cannot stall firing, and support snapshot() / load_snapshot() — a versioned, config-fingerprinted encoding of their dynamic state, which is what lets the engine checkpoint and restore them exactly.

Aggregations

AggOp: Count, Sum, Avg, Min, Max, CountDistinct, the variance and standard-deviation family (VarPop, VarSamp, StddevPop, StddevSamp, Welford online), boolean folds (BoolAnd, BoolOr) and bitwise folds. Build one with Agg::count("alias") or Agg::of(op, "column", "alias").

Part of FusionVault Streams, where the engine that drives these operators over Kafka lives. Apache-2.0.