Struct SemanticDht 

pub struct SemanticDht {
    pub shard_balancer: Mutex<ShardBalancer>,
    /* private fields */
}

Semantic DHT manager

Fields

shard_balancer: Mutex<ShardBalancer>

Shard-load balancer: tracks per-peer vector counts and identifies hot-spot peers that should shed load via migration.

Implementations

impl SemanticDht

pub fn new(config: SemanticDhtConfig) -> Self

Create a new semantic DHT

pub fn register_namespace( &self, namespace: SemanticNamespace, ) -> Result<(), SemanticDhtError>

Register a new semantic namespace

pub fn compute_lsh_hashes( &self, embedding: &[f32], namespace: &NamespaceId, ) -> Result<Vec<LshHash>, SemanticDhtError>

Compute LSH hashes for an embedding

pub fn index_content( &self, cid: Cid, embedding: Vec<f32>, namespace: NamespaceId, ) -> Result<(), SemanticDhtError>

Index content with its embedding

pub fn query( &self, query: SemanticQuery, ) -> Result<Vec<SemanticResult>, SemanticDhtError>

Execute a semantic query

pub fn stats(&self) -> SemanticDhtStats

Get statistics

pub fn get_namespace(&self, id: &NamespaceId) -> Option<SemanticNamespace>

Get namespace information

pub fn list_namespaces(&self) -> Vec<NamespaceId>

List all registered namespaces

pub fn put_with_vector( &self, cid: impl Into<String>, vector: Vec<f32>, provider_id: impl Into<String>, ) -> Result<(), SemanticDhtError>

Store a CID with its embedding vector in the local record store.

The record is validated (dimension check) and then written to the flat vector_records map. In a full network deployment the caller would subsequently gossip this record to neighbouring peers.

Errors

Returns SemanticDhtError::VectorDimensionMismatch when vector.len() != config.dimension.

pub fn search_similar( &self, query_vector: &[f32], k: usize, ) -> Result<Vec<(String, f32)>, SemanticDhtError>

Return the k CIDs whose stored vectors are most similar to query_vector.

Similarity is measured by cosine similarity (higher = more similar). Results are returned as (cid_string, similarity_score) pairs in descending score order.

Expired records (older than config.vector_ttl) are silently excluded.

Errors

Returns SemanticDhtError::VectorDimensionMismatch when query_vector.len() != config.dimension.

pub fn get_routing_convergence(&self) -> f32

Estimate how stable the routing table currently is.

Convergence is modelled as a function of how long the routing table has been quiescent: the score rises from 0 to 1 over 2 × sync_interval of silence. Once the score crosses convergence_threshold the table is considered converged.

Returns a value in [0.0, 1.0].

pub fn efficient_partial_sync( &self, peer: &PeerId, changed_region: &LshHash, ) -> Result<Vec<String>, SemanticDhtError>

Perform an efficient partial sync with a peer for a specific embedding region.

Only records whose vectors hash to changed_region (via LSH bucket equivalence) are considered. This avoids the cost of a full index exchange and is the primary mechanism for keeping distributed vector indices consistent between peers.

In the current single-node implementation the peer argument is validated and the region is used to filter the local record set; the function returns the set of CIDs that would be sent to the peer in a real network exchange.

Errors

Returns SemanticDhtError::PeerUnreachable when peer is the zero peer ID (used as a sentinel for “invalid peer” in tests).

pub fn efficient_partial_sync_with_config( &self, peer: &PeerId, changed_region: &LshHash, config: &PartialSyncConfig, prev_vectors: Option<&HashMap<String, Vec<f32>>>, ) -> Result<(Vec<String>, PartialSyncStats), SemanticDhtError>

Enhanced efficient partial sync with configurable threshold, batching, and round tracking.

Only gossips vectors whose embedding changed by more than config.sync_threshold (cosine distance) since the last known state stored in prev_vectors (a CID → previous vector snapshot).

Messages are batched up to config.batch_size per GossipSub batch. round_number is compared against config.max_rounds to detect divergence; if round_number >= config.max_rounds the caller should trigger a full resync (this function still executes normally but the returned stats will reflect the round count).

Returns the list of CIDs that would be sent plus sync statistics.

pub fn metrics(&self) -> SemanticDhtMetrics

Return a snapshot of routing-quality metrics.

pub fn rebalance_if_needed(&self) -> Vec<(String, String)>

Check whether any peer in the shard balancer is overloaded and, if so, return the list of (cid, from_peer) migration candidates.

Returns an empty Vec when the cluster is already balanced.

pub fn merge_partial_index( &self, records: Vec<VectorAnnotatedRecord>, source_peer: &str, ) -> MergeResult

Merge a partial index received from source_peer into the local record store.

For each incoming VectorAnnotatedRecord:

• Added – CID not yet known locally → insert.
• Updated – CID known but the incoming record has a strictly longer ttl_secs (a proxy for freshness when wall-clock timestamps are unavailable) → overwrite.
• Skipped – CID known and local copy is as fresh or fresher.
• Conflict – CID known but vector dimensions do not match → neither copy is overwritten; the conflict counter is incremented so the caller can log or raise an alert.

The source_peer argument is recorded in the provider_id field of any inserted/updated records, allowing provenance tracking.

pub fn evict_expired_records(&self)

Evict vector records that have exceeded their TTL.

Call this periodically (e.g. at sync_interval) to bound memory usage.

pub async fn distributed_search( &self, query: &[f32], top_k: usize, timeout_ms: u64, ) -> Result<Vec<SearchResult>, SemanticDhtError>

Query the semantic DHT across the local shard.

In a fully distributed deployment this method would fan out top_k sub-queries to each peer responsible for the relevant LSH bucket, merge the partial results, and apply timeout_ms. In the current single-node implementation it searches the local vector_records store and tags every result with peer = None (local).

Results are sorted by descending cosine similarity score.

Errors

Returns SemanticDhtError::VectorDimensionMismatch when query.len() != config.dimension.