Struct VectorCollection 

pub struct VectorCollection {
    pub surrogate_map: HashMap<u32, Surrogate>,
    pub surrogate_to_local: HashMap<Surrogate, u32>,
    pub multi_doc_map: HashMap<Surrogate, Vec<u32>>,
    pub codec_dispatch: Option<CollectionCodec>,
    pub payload: PayloadIndexSet,
    pub arena_index: Option<u32>,
    /* private fields */
}

Manages all vector segments for a single collection (one index key).

This type is !Send — owned by a single Data Plane core.

Fields

surrogate_map: HashMap<u32, Surrogate>

Mapping from internal global vector ID → surrogate.

surrogate_to_local: HashMap<Surrogate, u32>

Reverse map: surrogate → global vector ID. Used by point delete.

multi_doc_map: HashMap<Surrogate, Vec<u32>>

Reverse mapping for multi-vector documents: document_surrogate → list of global vector IDs.

codec_dispatch: Option<CollectionCodec>

Optional collection-level codec-dispatch index (RaBitQ or BBQ). Present only when the collection was built with a non-Sq8 quantization. Coexists with sealed segments — for codec-dispatched collections the per-segment Sq8 builder is skipped and this index is used instead.

payload: PayloadIndexSet

In-memory payload bitmap indexes for vector-primary collections.

Empty (no indexes) by default; populated at construction time from VectorPrimaryConfig::payload_indexes.

arena_index: Option<u32>

Optional dedicated memory arena index for this collection.

Set by the Data Plane after requesting a per-collection arena from nodedb_mem::CollectionArenaRegistry. Used only for stats reporting; the actual arena pinning is handled externally.

Implementations

impl VectorCollection

pub fn set_data_dir(&mut self, dir: PathBuf)

Set the data directory for mmap segment files.

pub fn set_ram_budget(&mut self, bytes: usize)

Set the RAM budget for vector data (FP32 in sealed segments).

pub fn ram_usage_bytes(&self) -> usize

Estimate current RAM usage for vector data.

pub fn is_budget_exceeded(&self) -> bool

Whether the RAM budget is exceeded.

pub fn mmap_fallback_count(&self) -> u32

Number of segments that fell back to mmap.

pub fn mmap_segment_count(&self) -> u32

Number of currently active mmap segments.

impl VectorCollection

pub fn building_segment_ids(&self) -> Vec<u32>

Segment ids of the building segments, oldest first.

pub fn build_request_for( &self, key: &str, segment_id: u32, ) -> Option<BuildRequest>

A seal build request for building segment segment_id, read from its flat vectors. None when no building segment has that id.

pub fn sealed_base_ids(&self) -> Vec<u32>

Base ids of the non-empty sealed segments, in segment order.

pub fn rebuild_request_for( &mut self, key: &str, base_id: u32, ) -> Result<Option<BuildRequest>, VectorError>

A rebuild request for the sealed segment at base_id, under the current HNSW params, read from that segment only: the growing and building segments keep their own vectors. None when no sealed segment starts at base_id.

Fails with VectorError::VectorUnavailable when a node’s vector cannot be read.

pub fn complete_build( &mut self, segment_id: u32, index: HnswIndex, memory: ScopedMemory, ) -> bool

Install a seal build: promote building segment segment_id to sealed with its current tombstones. Returns false, changing nothing, when no building segment has that id (a truncate dropped it) or the graph does not hold one node per vector of the segment.

pub fn complete_rebuild( &mut self, segment_id: u32, base_id: u32, len: usize, index: HnswIndex, memory: ScopedMemory, ) -> bool

Install a rebuild of the sealed segment at base_id, read when it held len nodes. The old segment’s tombstones carry over and the segment is quantized again under the collection’s config. Returns false, changing nothing, when no sealed segment at base_id still holds len nodes, or the graph does not hold len nodes.

pub fn note_build_failed(&mut self)

Record a build that failed. The segment stays as it was.

impl VectorCollection

pub fn checkpoint_to_bytes( &self, kek: Option<&WalEncryptionKey>, ) -> Result<Vec<u8>, VectorError>

Serialize all segments for checkpointing.

When kek is Some, the MessagePack payload is wrapped in an AES-256-GCM encrypted envelope with a SEGV preamble. When None, raw MessagePack bytes are returned (existing plaintext format).

Returns Err on serialization or encryption failure — callers must propagate rather than treat a failed encode as an empty checkpoint.

pub fn from_checkpoint( bytes: &[u8], kek: Option<&WalEncryptionKey>, memory: ScopedMemory, ) -> Result<Self, VectorError>

Restore a collection from checkpoint bytes.

kek controls the expected framing:

• None → the file must be plaintext MessagePack (starting with bytes that are NOT SEGV). If the file starts with SEGV and no key is provided, returns Err(CheckpointEncryptedNoKey).
• Some(key) → encryption is required. If the file starts with SEGV, it is decrypted with key. If the file is plaintext, returns Err(CheckpointPlaintextKeyRequired) — refuse to silently load unencrypted data when the operator has enabled at-rest encryption.

Returns Err(CheckpointDeserializationError) if the (decrypted) MessagePack payload, an embedded sealed-segment HNSW checkpoint, or an embedded PQ/SQ8 codec is structurally corrupt.

impl VectorCollection

pub fn build_codec_dispatch( &mut self, quantization: &str, ) -> Result<Option<&CollectionCodec>, VectorError>

Build a codec-dispatched index over the sealed vectors using the requested quantization. Replaces any existing dispatch index for this collection. Idempotent.

Returns a reference to the new index, or Ok(None) if the quantization tag is not supported (falls back to per-segment Sq8/PQ paths) or there are no vectors to train on.

impl VectorCollection

pub fn is_ivf(&self) -> bool

Whether the collection is configured as an IVF-PQ index.

pub fn index_config(&self) -> &IndexConfig

The full index configuration.

pub fn set_index_config(&mut self, config: IndexConfig)

Replace the index configuration. The HNSW parameters the collection holds stay: its segments were built with them.

pub fn ivf_training_threshold(&self) -> usize

Live vectors an IvfPq collection needs before it trains.

pub fn needs_ivf_training(&self) -> bool

Whether an IvfPq collection is untrained and holds the training threshold of live vectors.

pub fn ivf_index(&self) -> Option<&IvfPqIndex>

The trained IVF-PQ index, if any.

pub fn train_ivf( &mut self, memory: ScopedMemory, trained_at_ms: u64, ) -> Result<(), VectorError>

Train the IVF-PQ index on every live vector and move them all into it. trained_at_ms stamps the training, in Unix milliseconds.

Fails with VectorError::InvalidInput when the collection is not IvfPq, is already trained, or holds fewer live vectors than the training threshold. A training error propagates. Any failure leaves the collection unchanged.

impl VectorCollection

pub fn new(dim: usize, params: HnswParams) -> Self

Create an empty collection with the default seal threshold.

pub fn with_seal_threshold( dim: usize, params: HnswParams, seal_threshold: usize, ) -> Self

Create an empty collection with an explicit seal threshold.

pub fn with_index_config(dim: usize, config: IndexConfig) -> Self

Create an empty collection with a full index configuration.

pub fn with_seal_threshold_and_config( dim: usize, config: IndexConfig, seal_threshold: usize, ) -> Self

Create an empty collection with a full index config and custom seal threshold.

pub fn note_checkpoint_lsn(&mut self, lsn: u64)

Advance the applied-watermark to lsn if it is higher than the current value. Called at every apply chokepoint (live write and WAL replay) with the WAL LSN of the applied write. 0 is ignored — an unassigned LSN must never move the watermark. This does NOT move the replay skip gate (checkpoint_wal_lsn); it is folded into that gate only at checkpoint save time via max(checkpoint_wal_lsn, applied_wal_lsn), so the gate stays frozen across an entire replay pass and doesn’t gate a TransactionRedo record’s own siblings.

pub fn checkpoint_wal_lsn(&self) -> u64

Highest WAL LSN already reflected in this collection’s in-memory state, as of the last checkpoint load or save.

Startup replay skips any record whose lsn <= checkpoint_wal_lsn() for this collection: the restored checkpoint already contains that write, so re-applying it would append a duplicate HNSW node.

pub fn applied_wal_lsn(&self) -> u64

Highest WAL LSN applied since the last checkpoint load (the running max note_checkpoint_lsn builds). Folded into the persisted watermark at checkpoint save time; not consulted by the replay gate.

pub fn with_seed(dim: usize, params: HnswParams, _seed: u64) -> Self

Create with a specific seed (for deterministic testing).

pub fn set_seal_threshold(&mut self, threshold: usize)

Set the growing-segment size that triggers a seal. A zero threshold is raised to one vector.

pub fn needs_seal(&self) -> bool

Check if the growing segment should be sealed. An IvfPq collection never seals: its growing segment is the buffer IVF-PQ training reads.

pub fn seal(&mut self, key: &str) -> Option<BuildRequest>

Seal the growing segment and return a build request. None for an empty growing segment or an IvfPq collection.

pub fn sealed_segments(&self) -> &[SealedSegment]

Access sealed segments (read-only).

pub fn sealed_segments_mut(&mut self) -> &mut Vec<SealedSegment>

Access sealed segments mutably.

pub fn growing_is_empty(&self) -> bool

Whether the growing segment has no vectors.

pub fn len(&self) -> usize

pub fn live_count(&self) -> usize

pub fn is_empty(&self) -> bool

pub fn dim(&self) -> usize

pub fn params(&self) -> &HnswParams

pub fn set_params(&mut self, params: HnswParams)

Update HNSW parameters for future builds.

pub fn set_quantization(&mut self, q: VectorQuantization)

Set the collection-level quantization.

pub fn quantization(&self) -> VectorQuantization

Return the configured quantization mode.

pub fn configure_payload_indexes(&mut self, fields: &[String])

Configure payload bitmap indexes from a list of field names.

impl VectorCollection

pub fn truncate(&mut self) -> usize

Drop every vector, sealed segment, in-flight build, surrogate binding, and payload bitmap row. Returns the number of live vectors dropped.

Configuration survives: dimension, HNSW params, index config, quantization, seal threshold, memory budget, and the registered payload index fields. next_id and next_segment_id keep counting so a build completion for a segment sealed before the truncate finds no matching entry in building and is ignored, and an mmap file name is never reused. The mmap file of each dropped sealed segment is removed from disk.

An IvfPq collection drops its trained index too and buffers again: its next training reads the vectors inserted after the truncate.

pub fn compact(&mut self) -> usize

Compact sealed segments and the IVF-PQ index by removing tombstoned nodes.

Rewrites surrogate_map and multi_doc_map for every sealed segment so that global ids continue to resolve to the correct surrogate after local-id renumbering. IVF-PQ entries keep their ids.

pub fn multi_vector_documents(&self) -> Vec<Surrogate>

The multi-vector documents of this index, by document surrogate, in ascending order. export_snapshot rows carry no membership, so a snapshot records it here: a one-vector document is otherwise indistinguishable from a single-vector row.

pub fn export_snapshot(&self) -> Result<Vec<ExportedVector>, VectorError>

Export all live vectors for snapshot.

Errors

Propagates crate::HnswIndex::export_vectors if a sealed segment’s vectors cannot be materialized. A snapshot that silently dropped or zeroed those vectors would restore an index with missing data.

impl VectorCollection

pub fn insert(&mut self, vector: Vec<f32>) -> Result<u32, VectorError>

Insert a vector. Returns the global vector ID. A trained IVF-PQ collection inserts into its IVF index, every other collection into the growing segment.

A vector without the collection dimension fails with VectorError::DimensionMismatch and changes nothing.

pub fn insert_with_surrogate( &mut self, vector: Vec<f32>, surrogate: Surrogate, ) -> Result<u32, VectorError>

Insert a vector with an associated surrogate. The surrogate is allocated by the Control Plane before the call; the engine only stores the binding.

One surrogate names one live node. A node already bound to surrogate is soft-deleted before the new one is bound, so a re-insert never leaves an unreachable node scoring in searches. The caller owns the payload bitmap entries of the old node and removes them with Self::local_for_surrogate before this call.

A vector without the collection dimension fails with VectorError::DimensionMismatch before the old binding is touched.

Surrogate::ZERO binds nothing: a headless vector has no surrogate, so it never enters surrogate_map or surrogate_to_local, and no lookup or delete by ZERO reaches it. Compaction, checkpoint restore and rollback rebuild both maps from surrogate_map alone, so they never map ZERO either.

pub fn insert_batch_with_surrogates( &mut self, vectors: &[Vec<f32>], surrogates: &[Surrogate], ) -> Result<Vec<u32>, VectorError>

Insert a batch of vectors, the i-th bound to surrogates[i] (Surrogate::ZERO when the slice is shorter). Returns the global ids in batch order.

Every vector is checked against the collection dimension before any is inserted, so a mismatch fails with VectorError::DimensionMismatch and inserts none.

pub fn insert_multi_vector( &mut self, vectors: &[&[f32]], document_surrogate: Surrogate, ) -> Result<Vec<u32>, VectorError>

Insert multiple vectors for a single document (ColBERT-style). All N vectors are bound to the same document_surrogate.

Every vector is checked against the collection dimension before any is inserted, so a mismatch fails with VectorError::DimensionMismatch and inserts none.

pub fn delete_multi_vector(&mut self, document_surrogate: Surrogate) -> usize

Delete all vectors belonging to a multi-vector document.

pub fn get_surrogate(&self, vector_id: u32) -> Option<Surrogate>

Look up the surrogate for a global vector ID.

pub fn local_for_surrogate(&self, surrogate: Surrogate) -> Option<u32>

Resolve a surrogate back to its global vector ID, if bound.

pub fn delete(&mut self, id: u32) -> bool

Soft-delete a vector by global ID.

pub fn vector_for_id(&self, id: u32) -> Option<Vec<f32>>

The live FP32 vector stored under global id, whichever segment holds it. None for an unknown or soft-deleted id.

pub fn vector_for_surrogate(&self, surrogate: Surrogate) -> Option<Vec<f32>>

The live FP32 vector bound to surrogate, if any.

pub fn delete_by_surrogate(&mut self, surrogate: Surrogate) -> bool

Soft-delete a vector by surrogate.

pub fn undelete(&mut self, id: u32) -> bool

Un-delete a previously soft-deleted vector (for transaction rollback).

Symmetric to Self::delete_inner: the vector may live in the IVF index, the growing segment (the common case for a just-inserted vector), a sealed HNSW segment, or an in-flight building segment. The tombstone is reversed wherever it landed.

pub fn undelete_bound(&mut self, id: u32, surrogate: Surrogate) -> bool

Un-delete id and bind it to surrogate again: the reverse of Self::delete on a bound node, which drops the binding.

Returns false and changes nothing when id carries no tombstone. Surrogate::ZERO binds nothing, as in Self::insert_with_surrogate.

impl VectorCollection

pub fn hnsw_params(&self) -> HnswParams

Return the HNSW construction parameters for this collection.

pub fn growing_flat(&self) -> &FlatIndex

Immutable access to the growing flat index.

The growing index holds vectors that have not yet been sealed into an HNSW segment. Its contents should be included in any full-collection rebuild that wants to produce a complete result.

pub fn replace_sealed(&mut self, new_segments: Vec<SealedSegment>)

Replace all sealed segments with new_segments.

Used by the concurrent REINDEX cutover: after the background thread finishes rebuilding the HNSW graph the Data Plane swaps in the single rebuilt segment. The growing segment is preserved unchanged.

Caller is responsible for ensuring that new_segments covers all vectors that were in the old sealed set; any vector not present in the new segments will return no result on subsequent searches until the growing segment is sealed.

pub fn compact_tombstones(&mut self) -> usize

Compact tombstoned nodes from all sealed segments.

This is the same operation as compact() — the alias exists so call sites that want to express “remove tombstones” read clearly.

impl VectorCollection

pub fn with_pq_config(dim: usize, hnsw: HnswParams, pq_m: usize) -> Self

Convenience constructor for PQ-configured collections.

Equivalent to building a full IndexConfig with index_type = HnswPq and the given pq_m.

pub fn with_seal_threshold_and_pq_config( dim: usize, hnsw: HnswParams, pq_m: usize, seal_threshold: usize, ) -> Self

Convenience constructor for PQ-configured collections with a custom seal threshold.

pub fn build_sq8_for_index(index: &HnswIndex) -> Option<(Sq8Codec, Vec<u8>)>

Build SQ8 quantized data for an HNSW index.

Returns None when there are too few live vectors for stable min/max calibration.

pub fn build_pq_for_index( index: &HnswIndex, pq_m: usize, memory: ScopedMemory, ) -> Option<(PqCodec, Vec<u8>)>

Train a PQ codec from a built HNSW index’s live vectors, tracking codebook allocations against memory, and encode every node.

impl VectorCollection

pub fn is_live(&self, id: u32) -> bool

Whether node id exists and is not soft-deleted, whichever segment holds it.

pub fn write_mark( &self, surrogates: &[Surrogate], ids: &[u32], ) -> VectorWriteMark

Mark the state a write naming surrogates and node ids can change.

pub fn roll_back_to(&mut self, mark: VectorWriteMark) -> bool

Put the collection back to mark. Returns false, changing nothing, when a seal or an IVF-PQ training moved the nodes inserted since the mark out of the growing segment.

pub fn detached_empty(&self) -> Self

An empty collection with this one’s configuration and id counters: same dimension, parameters, index config, quantization, seal threshold, storage settings, registered payload fields and watermarks. The next insert takes the id this collection’s next insert would have taken.

impl VectorCollection

pub fn search( &self, query: &[f32], top_k: usize, ef: usize, ) -> Result<Vec<SearchResult>, VectorError>

Search across all segments, merging results by distance.

A query without the collection dimension fails with VectorError::DimensionMismatch before any segment is read.

pub fn search_with_metric( &self, query: &[f32], top_k: usize, ef: usize, metric: DistanceMetric, ) -> Result<Vec<SearchResult>, VectorError>

Search across all segments using an explicit metric override.

For sealed segments with quantized codecs, the metric override is applied during candidate reranking. Growing and building segments apply it exactly via brute-force. The HNSW graph structure was built with the collection metric; using a different metric affects the scoring but not graph traversal. A codec-dispatch index scores with the collection metric.

pub fn search_with_bitmap_bytes_and_metric( &self, query: &[f32], top_k: usize, ef: usize, bitmap: &[u8], metric: DistanceMetric, ) -> Result<Vec<SearchResult>, VectorError>

Search with a pre-filter bitmap (byte-array format) and explicit metric override.

Bitmap bytes that do not decode fail with VectorError::InvalidFilterBitmap.

pub fn search_with_bitmap_bytes( &self, query: &[f32], top_k: usize, ef: usize, bitmap: &[u8], ) -> Result<Vec<SearchResult>, VectorError>

Search with a pre-filter bitmap (byte-array format).

Bitmap bytes that do not decode fail with VectorError::InvalidFilterBitmap.

pub fn search_with_payload_filter( &self, query: &[f32], top_k: usize, ef: usize, predicate: &FilterPredicate, ) -> Result<(Vec<SearchResult>, bool), VectorError>

Search with a structured payload predicate.

If predicate is fully covered by indexed fields (all leaf fields have a bitmap index), the bitmap is built and HNSW traversal uses it as a pre-filter.

If any field in predicate is un-indexed, the method returns (results, false) where false signals that the predicate was NOT applied and the caller must apply it as a post-filter. This guarantees the un-indexed predicate is never silently dropped.

Returns (results, filter_was_applied).

impl VectorCollection

pub fn stats(&self) -> VectorIndexStats

Collect live statistics from all segments.