Struct ShardedColony

Source

pub struct ShardedColony { /* private fields */ }

Expand description

A sharded colony that participates in distributed coordination.

ShardedColony wraps a local Colony instance and adds the machinery needed for distributed operation:

Consistent hashing for document routing
Ghost node cache for cross-shard edge references
Cross-shard edge management for edges spanning shards
Peer shard address tracking
Cross-shard edge collection during tick phases

Implementations§

Source §

impl ShardedColony

Source

pub fn new( shard_id: ShardId, config: ColonyConfig, hash_ring: Arc<RwLock<ConsistentHashRing>>, ) -> Self

Create a new sharded colony.

§Arguments

shard_id - Unique identifier for this shard
config - Colony configuration parameters
hash_ring - Shared consistent hash ring for routing

§Example

let shard = ShardedColony::new(
    ShardId::new(0),
    ColonyConfig::default(),
    Arc::new(RwLock::new(ConsistentHashRing::new(3))),
);

Source

pub fn with_ghost_cache_size( shard_id: ShardId, config: ColonyConfig, hash_ring: Arc<RwLock<ConsistentHashRing>>, ghost_cache_size: usize, ) -> Self

Create a new sharded colony with a custom ghost cache size.

§Arguments

shard_id - Unique identifier for this shard
config - Colony configuration parameters
hash_ring - Shared consistent hash ring for routing
ghost_cache_size - Maximum number of ghost nodes to cache

Source

pub fn shard_id(&self) -> ShardId

Get this shard’s ID.

Source

pub async fn owns_document(&self, doc_id: &DocumentId) -> bool

Check if a document belongs to this shard.

Uses the consistent hash ring to determine ownership.

§Arguments

doc_id - The document ID to check

Source

pub fn owns_document_sync( &self, doc_id: &DocumentId, ring: &ConsistentHashRing, ) -> bool

Check ownership synchronously (for non-async contexts).

This is a blocking operation and should only be used when async is not available.

Source

pub async fn ingest_document( &mut self, title: &str, content: &str, position: Position, ) -> DistributedResult<DocumentId>

Ingest a document (only if this shard owns it).

Returns an error if the document should be routed to a different shard.

§Arguments

title - Document title
content - Document content
position - Spatial position in the substrate

§Returns

The document ID if successful, or a routing error if this shard doesn’t own the document.

Source

pub fn ingest_document_direct( &mut self, title: &str, content: &str, position: Position, ) -> DocumentId

Ingest a document with a pre-determined ID.

This is useful when routing a document from the coordinator, where the ID has already been assigned.

§Arguments

title - Document title
content - Document content
position - Spatial position in the substrate

Source

pub fn tick_phase(&mut self, phase: TickPhase) -> PhaseResult

Execute a tick phase.

Each tick is divided into phases that must be synchronized across all shards. The coordinator ensures all shards complete each phase before moving to the next.

§Arguments

phase - The phase to execute

§Returns

A PhaseResult containing statistics and any cross-shard edges that need resolution.

Source

pub fn get_term_frequencies(&self, terms: &[String]) -> HashMap<String, u64>

Get local term frequencies for TF-IDF computation.

This is called by the coordinator to aggregate document frequencies across all shards for proper TF-IDF scoring.

§Arguments

terms - The terms to count

§Returns

A map of term to document frequency on this shard.

Source

pub fn get_node(&self, id: &NodeId) -> Option<NodeData>

Get a node’s data from the local graph.

§Arguments

id - The node ID to look up

§Returns

The node data if found on this shard.

Source

pub fn add_peer(&mut self, shard_id: ShardId, address: String)

Add a peer shard address.

§Arguments

shard_id - The peer’s shard ID
address - The peer’s network address (e.g., “127.0.0.1:8081”)

Source

pub fn remove_peer(&mut self, shard_id: ShardId)

Remove a peer shard.

Also invalidates any ghost nodes from that shard.

§Arguments

shard_id - The peer’s shard ID

Source

pub fn peers(&self) -> &HashMap<ShardId, String>

Get peer addresses.

Source

pub fn peer_address(&self, shard_id: &ShardId) -> Option<&String>

Get a peer’s address.

Source

pub fn local(&self) -> &Colony

Get the local colony reference.

Source

pub fn local_mut(&mut self) -> &mut Colony

Get mutable local colony reference.

Source

pub fn ghost_cache(&self) -> &GhostNodeCache

Get ghost node cache.

Source

pub fn ghost_cache_mut(&mut self) -> &mut GhostNodeCache

Get mutable ghost node cache.

Source

pub fn edge_manager(&self) -> &CrossShardEdgeManager

Get the cross-shard edge manager.

Source

pub fn edge_manager_mut(&mut self) -> &mut CrossShardEdgeManager

Get mutable cross-shard edge manager.

Source

pub fn register_cross_shard_edge(&mut self, edge: CrossShardEdge)

Register an outgoing cross-shard edge with the edge manager.

This registers the edge for tracking and ghost node resolution. The edge is also added to pending_cross_edges for phase synchronization.

§Arguments

edge - The cross-shard edge to register

Source

pub fn handle_shard_offline(&mut self, shard_id: ShardId) -> (usize, usize)

Handle edges when a shard goes offline.

Removes all edges to/from the specified shard and invalidates corresponding ghost nodes.

§Arguments

shard_id - The shard that went offline

§Returns

A tuple of (edges_removed, ghosts_invalidated).

Source

pub fn decay_cross_shard_edges( &mut self, rate: f64, threshold: f64, ) -> Vec<CrossShardEdge>

Decay all cross-shard edges and prune weak ones.

§Arguments

rate - Decay rate (0.0 to 1.0)
threshold - Minimum weight threshold

§Returns

Vector of pruned edges.

Source

pub fn cross_shard_edge_stats(&self) -> CrossShardEdgeStats

Get statistics about cross-shard edges.

Source

pub fn connected_shards(&self) -> Vec<ShardId>

Get all shards this shard has edges to.

Source

pub fn take_pending_for_resolution( &mut self, ) -> HashMap<ShardId, Vec<CrossShardEdge>>

Resolve pending edges by fetching ghost nodes.

Takes the pending edges from the manager and returns them for processing. After ghost nodes are fetched, the caller should insert them into the ghost cache.