Skip to main content

Module semantic_dht

ipfrs_network

Module semantic_dht 

Source
Expand description

Semantic DHT - Vector-based content routing

This module extends the standard Kademlia DHT with semantic routing capabilities, allowing content discovery based on vector embeddings and semantic similarity rather than just content-addressed hashes.

§Features

  • Embedding-based Routing: Map vector embeddings to DHT keys using locality-sensitive hashing
  • Semantic Queries: Find content based on semantic similarity
  • Distributed ANN: Approximate nearest neighbor search across the network
  • Multiple Namespaces: Support different embedding spaces (text, image, etc.)
  • Adaptive Routing: Learn from query results to improve routing decisions

§Design

The semantic DHT uses a two-layer architecture:

  1. Embedding Layer: Maps high-dimensional embeddings to DHT keys via LSH
  2. Routing Layer: Routes queries using both XOR distance (Kademlia) and semantic distance

Each peer maintains:

  • A local vector index for semantic search
  • A mapping from embedding regions to peer IDs
  • Statistics on query success rates per region

Structs§

LshConfig
LSH configuration
LshHash
Locality-Sensitive Hash for embeddings
MergeResult
Result of a merge_partial_index operation.
NamespaceId
Semantic namespace identifier
PartialSyncConfig
Configuration for the efficient partial-sync algorithm (v0.3.0).
PartialSyncStats
Statistics collected during a single partial-sync pass.
SearchResult
A single result from SemanticDht::distributed_search.
SemanticDht
Semantic DHT manager
SemanticDhtConfig
Configuration for semantic DHT operations
SemanticDhtMetrics
Routing-quality metrics snapshot emitted by SemanticDht::metrics.
SemanticDhtStats
Statistics for semantic DHT
SemanticNamespace
Namespace configuration
SemanticQuery
Semantic DHT query
SemanticResult
Result from a semantic query
ShardBalancer
Tracks which peers own which HNSW-layer shards and computes load balance metrics used to drive migration decisions.
ShardBalancerConfig
Configuration for DHT shard balancing.
VectorAnnotatedRecord
A DHT record that carries an embedded vector alongside the CID it annotates.

Enums§

DistanceMetric
Distance metric for vector similarity
SemanticDhtError
Errors that can occur in semantic DHT operations