Module pipeline 

Stream processing pipeline module

This module provides a flexible, composable API for building and executing stream processing pipelines with support for:

• Fluent pipeline builder API
• Stream operators (map, filter, flatmap, key_by, window, aggregate)
• Event ingestion and processing
• Watermark propagation
• Window assignment and triggering
• State management
• Result emission

Example

use processor::pipeline::{StreamPipelineBuilder, StreamOperator};
use processor::config::{WindowConfig, StateBackend};
use std::time::Duration;

// Build a stream processing pipeline
let pipeline = StreamPipelineBuilder::new()
    .with_name("latency-aggregator")
    .with_parallelism(4)
    .with_window(WindowConfig::tumbling(60_000))
    .with_watermark_delay(Duration::from_secs(5))
    .with_state_backend(StateBackend::Memory)
    .build()?;

// Create an executor and run the pipeline
// let executor = pipeline.create_executor();
// executor.run().await?;

Re-exports

pub use builder::StreamPipelineBuilder;

pub use builder::StreamPipeline;

pub use builder::PipelineConfig;

pub use executor::StreamExecutor;

pub use executor::ExecutorStats;

pub use operator::StreamOperator;

pub use operator::MapOperator;

pub use operator::FilterOperator;

pub use operator::FlatMapOperator;

pub use operator::KeyByOperator;

pub use operator::WindowOperator;

pub use operator::AggregateOperator;

pub use operator::DeduplicationOperator;

pub use operator::OperatorContext;

pub use operator::DeduplicationStats;

pub use normalization_operator::NormalizationOperator;

pub use normalization_operator::NormalizationStats;

pub use normalization_operator::TimestampedEvent;

Modules

builder

Stream pipeline builder for fluent API configuration

executor

Stream executor for running stream processing pipelines

normalization_operator

Normalization operator for time-series data alignment

operator

Stream operators for transforming and processing events