Struct KafkaStreamPipeline 

pub struct KafkaStreamPipeline<T, R>
where
    T: Clone + Send + Sync + 'static,
    R: Clone + Send + Sync + 'static,
{
    pub executor: Arc<StreamExecutor<T>>,
    pub stats: Arc<RwLock<KafkaStreamStats>>,
    /* private fields */
}

End-to-end Kafka stream processing pipeline

This component orchestrates the complete Kafka-to-Kafka stream processing flow:

1. Consume events from input topics
2. Process through StreamExecutor (windowing, aggregation, etc.)
3. Produce results to output topic
4. Manage offsets automatically
5. Handle errors and recovery

Fields

executor: Arc<StreamExecutor<T>>

Stream executor (public for accessing outputs)

stats: Arc<RwLock<KafkaStreamStats>>

Statistics (public for monitoring)

Implementations

impl<T, R> KafkaStreamPipeline<T, R>

where T: Clone + Send + Sync + 'static + Serialize + for<'de> Deserialize<'de>, R: Clone + Send + Sync + 'static + Serialize + for<'de> Deserialize<'de>,

pub fn new( config: KafkaStreamConfig, executor: StreamExecutor<T>, ) -> Result<Self>

Create a new Kafka stream pipeline

pub async fn run( &mut self, result_rx: Receiver<ProcessorEvent<R>>, ) -> Result<()>

where T: EventTimeExtractor<T> + KeyExtractor<T>,

Run the complete pipeline

pub async fn shutdown(&mut self) -> Result<()>

Gracefully shutdown the pipeline

pub async fn stats(&self) -> KafkaStreamStats

Get current statistics

pub fn is_running(&self) -> bool

Check if pipeline is running

pub async fn health_check(&self) -> bool

Perform health check