Skip to main content

Module partial_sync

Module partial_sync 

Source
Expand description

Partial HNSW sync with dirty region tracking.

This module implements delta-based synchronisation of HNSW graph state between peers. Rather than shipping the entire index on every gossip round, it tracks which nodes have changed since the last successful sync and ships only those deltas.

§Architecture

┌──────────────────────────────────────────────────────────────┐
│  DirtyRegionTracker                                          │
│   ┌─────────────────────────────────────────────────────┐   │
│   │  dirty_nodes: HashMap<layer, HashSet<node_id>>      │   │
│   │  generation:  AtomicU64                             │   │
│   └─────────────────────────────────────────────────────┘   │
│                        ▲  ▼                                  │
│  PartialSyncManager                                          │
│   ┌─────────────────────────────────────────────────────┐   │
│   │  record_change / build_delta / apply_delta          │   │
│   │  pending_deltas: Vec<EmbeddingDelta> (ack tracking) │   │
│   └─────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────┘

Structs§

DirtyRegionTracker
Tracks which HNSW nodes are dirty (changed since the last successful sync).
EmbeddingDelta
A delta snapshot of changed embeddings in a region.
EmbeddingRegion
A region of the HNSW graph identified by layer + node ID range.
PartialSyncManager
Manages partial sync of HNSW graph changes between peers.