Struct DeferredIndexer 

pub struct DeferredIndexer { /* private fields */ }

Buffers vectors for deferred HNSW insertion with brute-force searchability.

See the module-level docs for design details.

Implementations

impl DeferredIndexer

pub fn new(config: DeferredIndexerConfig) -> Self

Creates a new DeferredIndexer with the given configuration.

The buffer starts inactive. If config.enabled is false, all write operations are no-ops.

pub fn is_enabled(&self) -> bool

Whether deferred indexing is enabled.

pub fn push(&self, id: u64, vector: Vec<f32>) -> bool

Pushes a vector into the write buffer.

Activates the buffer lazily on first write. Returns true if the buffer has reached merge_threshold, signaling the caller to trigger a merge.

No-op if deferred indexing is disabled.

TOCTOU note

The enabled check and len() >= threshold read are not atomic with the push. This is benign: a concurrent drain may reset the count between push and the threshold check, causing a missed merge signal. The next push will re-trigger.

A previous TOCTOU window existed where swap_and_drain could deactivate the buffer between ensure_buffer_active and the underlying buffer.push, causing the vector to be silently dropped. This is fixed: swap_and_drain now re-activates the buffer after draining so pushes between drain and the next merge succeed.

pub fn extend(&self, entries: impl IntoIterator<Item = (u64, Vec<f32>)>) -> bool

Batch-pushes vectors into the write buffer.

Returns true if the buffer has reached merge_threshold. No-op if deferred indexing is disabled.

pub fn remove(&self, id: u64)

Marks id as deleted, removing it from the buffer.

The ID is added to deleted_ids so that search results are filtered even if the vector was already snapshot for a concurrent search.

pub fn search( &self, query: &[f32], k: usize, metric: DistanceMetric, ) -> Vec<(u64, f32)>

Brute-force searches the buffer, filtering deleted IDs.

Results are sorted by the metric ordering and truncated to k.

To compensate for post-filter attrition, the buffer is queried with k + deleted_ids.len() candidates. This is bounded: deleted_ids never exceeds merge_threshold entries (cleared on every drain).

TOCTOU note

The deleted_ids snapshot is read under a separate lock from the buffer search. A concurrent delete between the buffer snapshot and the deleted_ids read is benign: the ID will be filtered on the next search after the delete completes.

pub fn merge_with_hnsw( &self, hnsw_results: Vec<(u64, f32)>, query: &[f32], k: usize, metric: DistanceMetric, ) -> Vec<(u64, f32)>

Merges HNSW results with deferred buffer results.

Buffer is authoritative on duplicate IDs (more recent data): when a point is upserted while deferred indexing is active, the new vector goes to the buffer while HNSW still holds the stale vector. On ID conflict the buffer score is kept, mirroring merge_with_delta in delta.rs.

Deleted IDs are filtered from buffer results but not from HNSW results (HNSW has its own tombstone system).

pub fn swap_and_drain(&self) -> Vec<(u64, Vec<f32>)>

Drains the buffer and returns vectors for HNSW insertion.

After this call the buffer is empty but re-activated so that pushes arriving between drain and the next merge are not silently dropped. The deleted_ids set is cleared because the caller is expected to apply deletions to HNSW after merge.

Serialized by an internal mutex so concurrent calls are safe (the second caller gets an empty drain).

pub fn pending_count(&self) -> usize

Total number of pending (not yet indexed) vectors in the buffer.

pub fn should_merge(&self) -> bool

Returns true if the buffer has reached merge_threshold.

pub fn is_searchable(&self) -> bool

Returns true if deferred indexing is enabled and the buffer has searchable data.

pub fn drain_all(&self) -> Vec<(u64, Vec<f32>)>

Drains all vectors from the buffer (for shutdown / flush).

Clears deleted_ids. After this call the buffer is empty and inactive.