Module graph 

Graph-Based Execution for StreamWeave

This module provides comprehensive graph-based execution capabilities for StreamWeave, enabling producers, transformers, and consumers to be connected in complex graph topologies for sophisticated data processing pipelines.

Overview

The graph module provides:

• Graph Construction: Type-safe graph builders with compile-time validation
• Complex Topologies: Support for fan-in, fan-out, and complex routing patterns
• Node System: Wrapper nodes for producers, transformers, and consumers
• Execution Engine: Concurrent execution of graph nodes with stream routing
• Zero-Copy Support: Efficient zero-copy data sharing for in-process execution
• Distributed Execution: Support for distributed graph execution across nodes
• Serialization: Message serialization for distributed execution
• Routing: Flexible routing strategies (round-robin, broadcast, custom)

Core Components

• Graph: The main graph structure for executing data processing pipelines
• GraphBuilder: Type-safe builder for constructing graphs with compile-time validation
• RuntimeGraphBuilder: Dynamic builder for runtime graph construction
• Nodes: Wrapper types for producers, transformers, and consumers in graphs
• Execution: Execution engine for running graphs with concurrent node execution
• Router: Routing strategies for distributing data across multiple outputs
• Channels: Communication channels between graph nodes

Universal Message Model

All data flowing through graphs is automatically wrapped in Message<T>. This ensures message IDs, metadata, and error correlation are preserved throughout the graph execution.

Example

use streamweave::graph::{Graph, GraphBuilder};
use streamweave::producers::VecProducer;
use streamweave::transformers::MapTransformer;
use streamweave::consumers::VecConsumer;

let graph = GraphBuilder::new()
    .add_producer("source", VecProducer::new(vec![1, 2, 3]))
    .add_transformer("transform", MapTransformer::new(|x: i32| x * 2))
    .add_consumer("sink", VecConsumer::new())
    .connect("source", "transform")
    .connect("transform", "sink")
    .build();

Re-exports

pub use channels::*;

pub use connection::*;

pub use execution::*;

pub use graph::*;

pub use graph_builder::*;

pub use http_server::*;

pub use nodes::*;

pub use router::*;

pub use serialization::*;

pub use shared_memory_channel::*;

pub use subgraph::*;

pub use tcp_server::*;

pub use throughput::*;

pub use traits::*;

pub use zero_copy::*;

Modules

channels

Type-Erased Zero-Copy Channels for Message<T>

connection

Connection Types

execution

Graph Execution Engine

graph

Graph Structure

graph_builder

Graph Builder

http_server

HTTP Server Integration for StreamWeave Graphs

nodes

Graph Node Types

router

Router Traits

serialization

Serialization utilities for graph node execution.

shared_memory_channel

Shared Memory Channels

stateful

Stateful Node Support

subgraph

Subgraph Support

tcp_server

TCP Server Module for StreamWeave Graphs

throughput

Throughput Monitoring Module

traits

Graph Traits

windowing

Graph Windowing Operations

zero_copy

Zero-copy architecture traits and utilities.