Expand description
§Rivven Cluster
Distributed clustering for Rivven with:
- SWIM Protocol: Scalable membership and failure detection
- Raft Consensus: Metadata coordination and leader election
- ISR Replication: Partition data replication with high watermarks
§Architecture
┌─────────────────────────────────────────────────────────────┐
│ Rivven Node │
├──────────────┬──────────────┬───────────────────────────────┤
│ SWIM │ Raft │ ISR Replication │
│ Membership │ Metadata │ Data Layer │
├──────────────┼──────────────┼───────────────────────────────┤
│ • Node disco │ • Topics │ • Partition leaders │
│ • Health chk │ • Partitions │ • Follower fetch │
│ • Failure │ • Assignments│ • High watermark │
│ detection │ • ISR state │ • Catch-up sync │
└──────────────┴──────────────┴───────────────────────────────┘§Deployment Modes
- Standalone: Single node, zero configuration, all partitions local
- Cluster: Multi-node with SWIM membership, Raft metadata, ISR replication
§Example Usage
ⓘ
use rivven_cluster::{ClusterConfig, ClusterCoordinator, TopicConfig};
// Standalone mode (zero config)
let config = ClusterConfig::standalone();
let coordinator = ClusterCoordinator::standalone(config).await?;
// Create a topic
coordinator.create_topic(TopicConfig::new("events", 6, 1)).await?;
// Cluster mode
let config = ClusterConfig::cluster()
.node_id("node-1")
.seeds(vec!["node-2:9093", "node-3:9093"])
.build();
let mut coordinator = ClusterCoordinator::new(config).await?;
coordinator.start().await?;Re-exports§
pub use config::ClusterConfig;pub use config::ClusterMode;pub use config::RaftConfig;pub use config::ReplicationConfig;pub use config::SwimConfig;pub use config::TopicDefaults;pub use consumer_coordinator::ConsumerCoordinator;pub use consumer_coordinator::CoordinatorError;pub use consumer_coordinator::CoordinatorResult;pub use coordinator::ClusterCoordinator;pub use coordinator::ClusterHealth;pub use coordinator::CoordinatorState;pub use error::ClusterError;pub use error::Result;pub use membership::Membership;pub use membership::MembershipEvent;pub use membership::SwimMessage;pub use metadata::ClusterMetadata;pub use metadata::MetadataCommand;pub use metadata::MetadataResponse;pub use metadata::MetadataStore;pub use node::Node;pub use node::NodeCapabilities;pub use node::NodeId;pub use node::NodeInfo;pub use node::NodeState;pub use observability::init_metrics;pub use observability::ClusterMetrics;pub use observability::NetworkMetrics;pub use observability::RaftMetrics;pub use partition::PartitionId;pub use partition::PartitionState;pub use partition::TopicConfig;pub use partition::TopicState;pub use placement::PartitionPlacer;pub use placement::PlacementConfig;pub use placement::PlacementStrategy;pub use protocol::Acks;pub use protocol::ClusterRequest;pub use protocol::ClusterResponse;pub use raft::hash_node_id;pub use raft::LogStore;pub use raft::NetworkFactory as RaftNetworkFactory;pub use raft::RaftController;pub use raft::RaftNode;pub use raft::RaftNodeConfig;pub use raft::RaftNodeId;pub use raft::StateMachine as MetadataStateMachine;pub use raft::TypeConfig as RaftTypeConfig;pub use replication::PartitionReplication;pub use replication::ReplicationManager;pub use storage::RedbLogStore;pub use transport::Transport;pub use transport::TransportConfig;
Modules§
- config
- Cluster configuration
- consumer_
coordinator - coordinator
- Cluster coordinator - orchestrates all cluster components
- error
- Cluster error types
- membership
- SWIM protocol implementation for cluster membership
- metadata
- Cluster metadata and Raft state machine
- node
- Node types and management
- observability
- Unified observability layer for Rivven cluster
- partition
- Partition types and state management
- placement
- Partition placement using consistent hashing with rack awareness
- prelude
- Re-export common types
- protocol
- Wire protocol for cluster communication
- raft
- Raft integration for cluster metadata consensus
- replication
- ISR (In-Sync Replica) replication implementation
- storage
- Pluggable storage backends for Raft log persistence
- transport
- Network transport for cluster communication