Module distributed 

Distributed Stream Processing

Provides infrastructure for coordinating stream processing across multiple nodes in a cluster. This module implements consistent hashing for partition routing, distributed window aggregation, and cluster-wide job distribution.

Components

• DistributedStreamTopology: Coordinates stream processing across nodes
• ConsistentHashRouter: Routes stream partitions to nodes via consistent hashing
• DistributedWindowAggregator: Aggregates windowed results across cluster nodes
• ClusterStreamCoordinator: Manages stream job distribution across the cluster

W3-S11 (Flink-lite distributed shards via Raft)

• coordinator::DistributedStreamCoordinator: Raft-tracked shard assignment, per-key routing, integration with the W3-S9 cluster sink for assignment durability.
• shard_manager::ShardManager: deterministic balanced shard placement and rebalance plan generation.
• event_shipper::EventShipper: cross-shard event delivery with a pluggable transport.

Re-exports

pub use coordinator::CoordinatorConfig;

pub use coordinator::CoordinatorError;

pub use coordinator::CoordinatorResult;

pub use coordinator::CoordinatorStats as DistributedCoordinatorStats;

pub use coordinator::CoordinatorStatsSnapshot as DistributedCoordinatorStatsSnapshot;

pub use coordinator::DistributedStreamCoordinator;

pub use coordinator::RoutedEvent;

pub use event_shipper::EventShipper;

pub use event_shipper::InProcessShipperTransport;

pub use event_shipper::ShippedEvent;

pub use event_shipper::ShipperConfig;

pub use event_shipper::ShipperError;

pub use event_shipper::ShipperResult;

pub use event_shipper::ShipperStats;

pub use event_shipper::ShipperStatsSnapshot;

pub use event_shipper::ShipperTransport;

pub use shard_manager::NodeId as ShardManagerNodeId;

pub use shard_manager::RebalancePlan;

pub use shard_manager::ShardAssignment;

pub use shard_manager::ShardId;

pub use shard_manager::ShardManager;

pub use shard_manager::ShardManagerConfig;

pub use shard_manager::ShardManagerError;

pub use shard_manager::ShardManagerResult;

pub use shard_manager::ShardMove;

Modules

coordinator

Distributed stream coordinator

event_shipper

Event shipper

shard_manager

Shard manager

Structs

AggregatedWindowResult

Aggregated result across all nodes for a window

ClusterStreamCoordinator

Manages stream job distribution across a cluster.

ConsistentHashRouter

Routes stream partitions to nodes using consistent hashing.

CoordinatorStats

Statistics for the cluster coordinator

DistributedStreamTopology

Coordinates stream processing across multiple cluster nodes.

DistributedWindowAggregator

Aggregates windowed results from multiple cluster nodes.

PartialWindowResult

A partial window result from a single node

StreamJob

A stream processing job that is distributed across cluster nodes

StreamNode

Metadata for a node participating in distributed stream processing

TopologyConfig

Configuration for a distributed stream topology

TopologyStats

Statistics snapshot for the topology

Enums

DistributedStreamError

Errors that can occur in distributed stream processing

NodeStatus

Represents the health status of a cluster node

TopologyChange

A topology change event broadcast on topology changes

Type Aliases

DistributedResult

Result type for distributed stream operations