fv-streams-ops 0.3.0

Stateful streaming operators for FusionVault Streams: event-time tumbling and sliding windows with keyed incremental aggregation, session windows, stream joins, top-N and last-N, watermarks and snapshots. Pure and synchronous — no I/O, no runtime.
Documentation
//! Per-partition event-time watermark — the progress signal every stateful operator uses to decide
//! when a window/session may fire or a buffered join row may be evicted.
//!
//! The rule (shared by [`WindowAgg`](crate::WindowAgg), [`SessionAgg`](crate::SessionAgg) and
//! [`StreamJoin`](crate::StreamJoin)): a consumer may read several partitions, so the operative
//! watermark is the **min across partitions** of each partition's max event time, minus an
//! `allowed_lateness` grace. Taking the min means a lagging (or not-yet-polled) partition holds firing
//! back, so a consumer never fires ahead of its slowest input — the standard multi-input watermark.
//! [`expect`](Watermark::expect) seeds assigned-but-unseen partitions so the min stays `i64::MIN` until
//! every assigned partition has produced an event.
//!
//! ## Bounded idleness
//!
//! A genuinely IDLE assigned partition (an empty partition, or one whose producer has gone quiet) would
//! otherwise pin the watermark forever — nothing fires until shutdown. With an idle timeout configured
//! ([`with_idle_timeout`](Watermark::with_idle_timeout)), a partition that has not advanced its max
//! event time for longer than the timeout (measured in **processing time** — wall-clock ms, which the
//! caller supplies; event time can't measure the *absence* of events) is treated as IDLE and dropped
//! from the min, so the active partitions drive progress. It rejoins the moment it produces again. The
//! processing-time clock is injected by the caller (`now_ms` on [`expect`](Watermark::expect),
//! [`tick`](Watermark::tick), [`current`](Watermark::current)), keeping this module pure and
//! deterministic — a test drives idleness with synthetic clock values. When ALL partitions are idle the
//! watermark holds (returns `i64::MIN`): with no active input there is no safe progress signal, and a
//! true end-of-stream is handled by the operator's `flush`. Idleness is opt-in; with no timeout the
//! watermark is exactly the min over all seen/assigned partitions (`now_ms` is then ignored).

use std::collections::HashMap;

/// Tracks the max event time seen per partition and derives the min-across-partitions watermark, with
/// optional bounded-idleness so a silent partition stops holding progress back.
pub(crate) struct Watermark {
    allowed_lateness_ms: i64,
    /// `Some(ms)` enables bounded idleness: a partition silent for longer than `ms` (processing time)
    /// drops out of the min. `None` (default) disables it — the min is over all seen/assigned partitions.
    idle_timeout_ms: Option<i64>,
    /// Per-partition max event time.
    part_max: HashMap<i32, i64>,
    /// Processing-time (wall ms) of each partition's last *advance* of its max, or its assignment for a
    /// not-yet-seen partition. Updated by [`tick`](Self::tick); drives idle detection.
    last_activity: HashMap<i32, i64>,
    /// `part_max` snapshot at the last [`tick`](Self::tick), to detect which partitions advanced.
    last_seen_max: HashMap<i32, i64>,
}

impl Watermark {
    pub fn new(allowed_lateness_ms: i64) -> Self {
        Self {
            allowed_lateness_ms: allowed_lateness_ms.max(0),
            idle_timeout_ms: None,
            part_max: HashMap::new(),
            last_activity: HashMap::new(),
            last_seen_max: HashMap::new(),
        }
    }

    /// Enable bounded idleness with a processing-time timeout (ms). `<= 0` leaves it disabled.
    pub fn with_idle_timeout(mut self, idle_timeout_ms: i64) -> Self {
        self.idle_timeout_ms = (idle_timeout_ms > 0).then_some(idle_timeout_ms);
        self
    }

    /// Register the partitions this operator will read, at processing time `now_ms`, so the watermark
    /// holds until each has reported. A brand-new partition seeds its max at `i64::MIN` and starts its
    /// idle timer at `now_ms` (so an assigned-but-never-produced partition eventually goes idle rather
    /// than pinning the watermark forever). Idempotent — an already-known partition is untouched.
    pub fn expect(&mut self, partitions: &[i32], now_ms: i64) {
        for &p in partitions {
            self.part_max.entry(p).or_insert(i64::MIN);
            self.last_activity.entry(p).or_insert(now_ms);
            self.last_seen_max.entry(p).or_insert(i64::MIN);
        }
    }

    /// Record an event's time on its partition (monotonic max per partition). Cheap and clock-free — the
    /// idle timer is advanced separately by [`tick`](Self::tick) once per batch.
    pub fn observe(&mut self, partition: i32, event_ms: i64) {
        let pm = self.part_max.entry(partition).or_insert(i64::MIN);
        if event_ms > *pm {
            *pm = event_ms;
        }
    }

    /// Advance the idle timers at processing time `now_ms`: any partition whose max advanced since the
    /// last tick has its activity stamped `now_ms`; the rest keep their previous stamp (and so age
    /// toward idle). Call once per batch before reading [`current`](Self::current). A no-op for idle
    /// detection when idleness is disabled, but kept cheap either way.
    pub fn tick(&mut self, now_ms: i64) {
        let snapshot: Vec<(i32, i64)> = self.part_max.iter().map(|(&p, &m)| (p, m)).collect();
        for (p, pm) in snapshot {
            let seen = self.last_seen_max.entry(p).or_insert(i64::MIN);
            if pm > *seen {
                *seen = pm;
                self.last_activity.insert(p, now_ms);
            } else {
                // Brand-new partition first seen via `observe` (never `expect`ed): start its timer now.
                self.last_activity.entry(p).or_insert(now_ms);
            }
        }
    }

    /// The current watermark at processing time `now_ms`: `min over ACTIVE partitions(max event time) −
    /// allowed-lateness`. With idleness disabled, "active" is every seen/assigned partition (so this is
    /// the plain min and `now_ms` is irrelevant). With idleness enabled, a partition silent longer than
    /// the timeout is excluded. `i64::MIN` when no partition is active (none seen, or all idle).
    pub fn current(&self, now_ms: i64) -> i64 {
        let active_min = self
            .part_max
            .iter()
            .filter(|(p, _)| self.is_active(**p, now_ms))
            .map(|(_, &m)| m)
            .min();
        match active_min {
            None => i64::MIN,
            Some(m) => m.saturating_sub(self.allowed_lateness_ms),
        }
    }

    /// A watermark delivered from upstream (already the minimum across its inputs), with this
    /// operator's allowed lateness applied — the time an operator fires by when its time comes
    /// in band instead of from its own partitions' maxima.
    pub fn after_lateness(&self, watermark_ms: i64) -> i64 {
        if watermark_ms == i64::MIN {
            i64::MIN
        } else {
            watermark_ms.saturating_sub(self.allowed_lateness_ms)
        }
    }

    fn is_active(&self, partition: i32, now_ms: i64) -> bool {
        match self.idle_timeout_ms {
            None => true,
            Some(timeout) => {
                let last = self.last_activity.get(&partition).copied().unwrap_or(now_ms);
                now_ms - last <= timeout
            }
        }
    }

    // ── snapshot/restore: only the DYNAMIC maps; lateness/idle-timeout are config (constructor). ──

    /// The watermark's contribution to an operator's config fingerprint (restore refuses a change).
    pub(crate) fn cfg_fingerprint(&self) -> String {
        format!("lateness={};idle={:?}", self.allowed_lateness_ms, self.idle_timeout_ms)
    }

    pub(crate) fn state_to_json(&self) -> serde_json::Value {
        serde_json::json!({
            "part_max": crate::snapshot::i32map_to_json(&self.part_max),
            "last_activity": crate::snapshot::i32map_to_json(&self.last_activity),
            "last_seen_max": crate::snapshot::i32map_to_json(&self.last_seen_max),
        })
    }

    pub(crate) fn load_state(&mut self, j: &serde_json::Value) -> Result<(), String> {
        self.part_max =
            crate::snapshot::i32map_from_json(&j["part_max"]).map_err(|e| format!("watermark part_max: {e}"))?;
        self.last_activity = crate::snapshot::i32map_from_json(&j["last_activity"])
            .map_err(|e| format!("watermark last_activity: {e}"))?;
        self.last_seen_max = crate::snapshot::i32map_from_json(&j["last_seen_max"])
            .map_err(|e| format!("watermark last_seen_max: {e}"))?;
        Ok(())
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn min_across_partitions_holds_back_the_faster_one() {
        let mut w = Watermark::new(0);
        w.observe(0, 1000);
        w.observe(1, 400);
        assert_eq!(w.current(0), 400, "the lagging partition sets the watermark");
        w.observe(1, 1200);
        assert_eq!(w.current(0), 1000, "now partition 0 is the laggard");
    }

    #[test]
    fn expected_but_unseen_partition_pins_at_min() {
        let mut w = Watermark::new(0);
        w.expect(&[0, 1], 0);
        w.observe(0, 5000);
        assert_eq!(w.current(0), i64::MIN, "partition 1 assigned but silent → no progress");
        w.observe(1, 100);
        assert_eq!(w.current(0), 100);
    }

    #[test]
    fn allowed_lateness_shifts_the_watermark_back() {
        let mut w = Watermark::new(250);
        w.observe(0, 1000);
        assert_eq!(w.current(0), 750);
    }

    #[test]
    fn empty_is_min() {
        assert_eq!(Watermark::new(0).current(0), i64::MIN);
    }

    #[test]
    fn idleness_disabled_ignores_the_clock() {
        // No idle timeout: a long-silent partition still pins the min forever (today's behaviour).
        let mut w = Watermark::new(0);
        w.expect(&[0, 1], 0);
        w.observe(0, 5000);
        w.tick(1_000_000); // huge processing-time jump — but idleness is off
        assert_eq!(w.current(1_000_000), i64::MIN, "silent partition 1 still holds");
    }

    #[test]
    fn a_silent_partition_goes_idle_and_releases_the_watermark() {
        let mut w = Watermark::new(0).with_idle_timeout(1000);
        w.expect(&[0, 1], 0);
        w.observe(0, 5000);
        w.observe(1, 400);
        w.tick(0);
        assert_eq!(w.current(0), 400, "both active → min holds at the laggard");
        // Partition 0 keeps producing (stays active); partition 1 goes quiet and ages out.
        w.observe(0, 6000);
        w.tick(500);
        assert_eq!(w.current(500), 400, "within timeout, partition 1 still holds");
        w.observe(0, 7000); // partition 0 advances again at processing-time 1600 → stays active
        w.tick(1600); // partition 1 silent since t=0 → 1600 > 1000 timeout → idle
        assert_eq!(
            w.current(1600),
            7000,
            "idle partition 1 dropped; partition 0 (still producing) drives"
        );
    }

    #[test]
    fn an_assigned_but_never_seen_partition_goes_idle() {
        let mut w = Watermark::new(0).with_idle_timeout(1000);
        w.expect(&[0, 1], 0); // partition 1 never produces
        w.observe(0, 5000);
        w.tick(0);
        assert_eq!(
            w.current(0),
            i64::MIN,
            "partition 1 assigned, timer not yet expired → holds"
        );
        w.observe(0, 5001); // partition 0 keeps producing → stays active past the timeout
        w.tick(1600); // 1600 > 1000 since assignment → partition 1 idle
        assert_eq!(
            w.current(1600),
            5001,
            "never-seen partition released after its idle timeout"
        );
    }

    #[test]
    fn an_idle_partition_rejoins_when_it_produces_again() {
        let mut w = Watermark::new(0).with_idle_timeout(1000);
        w.expect(&[0, 1], 0);
        w.observe(0, 5000);
        w.tick(2000); // partition 1 idle (silent since assignment)
        assert_eq!(w.current(2000), 5000);
        w.observe(1, 300); // partition 1 speaks again — a laggard, but now active
        w.tick(2001);
        assert_eq!(
            w.current(2001),
            300,
            "rejoined partition holds the watermark back again"
        );
    }

    #[test]
    fn all_idle_holds_at_min() {
        let mut w = Watermark::new(0).with_idle_timeout(1000);
        w.expect(&[0, 1], 0);
        w.observe(0, 5000);
        w.observe(1, 6000);
        w.tick(0);
        w.tick(5000); // both silent since t=0 → both idle
        assert_eq!(w.current(5000), i64::MIN, "no active partition → no safe progress");
    }
}