Struct EmbeddingIndexMerger 

pub struct EmbeddingIndexMerger {
    pub config: MergeConfig,
}

Merges multiple partial HNSW embedding index shards into a unified index.

Example

use ipfrs_semantic::index_merger::{
    EmbeddingIndexMerger, IndexShard, MergeConfig, ShardEntry,
};

let config = MergeConfig::default();
let merger = EmbeddingIndexMerger::new(config);

let shard = IndexShard::new(
    "node-1",
    vec![ShardEntry::new(1, vec![1.0_f32, 0.0], "doc-a")],
    2,
);

let (entries, stats) = merger.merge(&[shard]).expect("merge failed");
assert_eq!(entries.len(), 1);
assert_eq!(stats.output_entries, 1);

Fields

config: MergeConfig

Configuration for this merger instance.

Implementations

impl EmbeddingIndexMerger

pub fn new(config: MergeConfig) -> Self

Creates a new EmbeddingIndexMerger with the supplied configuration.

pub fn merge( &self, shards: &[IndexShard], ) -> Result<(Vec<ShardEntry>, MergeStats), MergeConflict>

Merges the provided shards into a single flat list of ShardEntry values.

Algorithm

1. Each shard is validated; a dimension mismatch causes an immediate Err.
2. Entries are walked in shard order. A HashMap<u64, usize> maps each seen embedding ID to its current position in the output vector.
3. On a duplicate ID the cosine distance between the stored vector and the incoming vector is computed:
   • Distance > conflict_threshold → MergeConflict::DuplicateId is appended to stats.conflicts; the new entry is always skipped.
   • Distance ≤ conflict_threshold → near-identical copy; skipped silently.
   • When keep_first = false the existing slot is overwritten regardless of whether the conflict was recorded.
4. If adding a new (non-duplicate) entry would push the output beyond max_entries, Err(CapacityExceeded) is returned immediately.

Returns (merged_entries, stats) on success.

pub fn cosine_distance(a: &[f32], b: &[f32]) -> f32

Computes the cosine distance between two vectors.

cosine_distance = 1.0 - cosine_similarity

Returns 1.0 when either vector has zero norm (undefined similarity).

pub fn shard_coverage(shards: &[IndexShard]) -> Vec<(String, usize)>

Returns a list of (shard_id, entry_count) pairs sorted by entry count in descending order.