Module watermark 

Watermarking system for handling late events in stream processing

This module provides watermarking capabilities to handle out-of-order and late-arriving events in the stream processing pipeline. Watermarks represent a timestamp threshold indicating that all events with timestamps below this threshold have been processed.

Overview

The watermarking system supports:

• Bounded out-of-orderness handling
• Per-partition watermark tracking
• Idle partition detection
• Periodic and punctuated watermark generation
• Late event detection and handling

Example

use processor::watermark::{Watermark, BoundedOutOfOrdernessWatermark, WatermarkGenerator};
use std::time::Duration;

let mut generator = BoundedOutOfOrdernessWatermark::new(
    Duration::from_secs(10),  // 10 second max out-of-orderness
    Some(Duration::from_secs(60)) // 60 second idle timeout
);

// Process an event with timestamp 1000ms
if let Some(watermark) = generator.on_event(1000, 0) {
    println!("New watermark: {:?}", watermark);
}

Structs

BoundedOutOfOrdernessConfig

Configuration for bounded out-of-orderness watermarking

BoundedOutOfOrdernessWatermark

Bounded out-of-orderness watermark generator

LateEventConfig

Late event handler configuration

LateEventHandler

Handler for late events

LateEventStats

Statistics for late event handling

PeriodicWatermark

Periodic watermark generator

PeriodicWatermarkConfig

Configuration for periodic watermark emission

PunctuatedWatermark

Punctuated watermark generator

Watermark

Represents a watermark timestamp

Traits

WatermarkGenerator

Trait for generating watermarks from event streams