datum-cluster-sharding

datum-cluster-sharding is the v0.10 cluster-sharding satellite for Datum. It ships ShardEnvelope, shard extraction, deterministic oldest-member coordinator allocation, peer ShardRegion routing over C2 DCP node-session shard pipes, bounded buffering while a shard is allocated, bounded handoff buffering while a shard is rebalanced, and first-message entity actor startup. The data path uses long-lived per-peer pipes with batch-when-behind/no-linger flushing; shard allocation misses remain on the control-plane request/response path.

Delivery in v0.10 is intentionally modest: at-most-once per send attempt, ordering per entity from a single sender that awaits each send, in-memory allocation tables, and no persistent shard allocation store. When a node is downed, the coordinator reallocates its shards with the least-shards strategy; entity state is not migrated. Graceful join-driven rebalance uses the same strategy with rebalance_per_round as the Akka-style per-round cap.

Remember-entities is opt-in per entity type via register_remembered_entity_type. A configured RememberEntitiesStore records entity starts/stops through a bounded write-behind queue, so store writes do not block normal message delivery. When a remembered shard starts locally after rebalance, restart recovery, or coordinator takeover, the region lists the shard's remembered ids and starts those entity actors without waiting for the next message.

Two stores ship with the SPI. InMemoryStore is for tests and single-process development: it survives rebalance only when all regions share the same store value, and it does not survive process death. FileStore writes one append log per shard under a configured directory and compacts the shard log on passivation. Its default fsync policy fsyncs appends and compactions, but delivery is still write-behind: a message can be acknowledged before the start record is durable. FileStoreFsyncPolicy::Never and OnCompaction trade durability for less storage overhead. A future datum-cdc/persistent store is a follow-up, not part of this crate.

Passivation removes remembered ids: EntityRef::passivate() stops the current entity actor and deletes it from the remember store, while passivation_idle_timeout passivates idle local entities on the region tick. If a passivation command or user envelope reaches a moving shard, it is ordered through the same bounded handoff buffer before the new owner sees later messages. Ordinary entity actor failure/restart and node stop are not passivation and do not remove the id from the store.

Coordinator takeover rebuilds the allocation table from surviving regions and keeps the highest generation for each shard. Any rebuilt entry that still points at a non-eligible member is reallocated by the new coordinator, so an in-flight migration can be lost as a generation update but cannot orphan the shard. Remember store failures publish RememberEntitiesEvent shard-level errors. FailOpen keeps routing after an event; FailClosed marks that shard failed so future messages fail fast until the region is restarted or replaced.