Struct VectorCollection 

pub struct VectorCollection { /* private fields */ }

A vector collection combining HNSW search, payload storage, and full-text search.

VectorCollection is a typed newtype over Collection that provides a stable public API for vector workloads. All storage operations delegate to the single inner: Collection instance — no dual-storage desync.

Examples

use velesdb_core::{VectorCollection, DistanceMetric, Point, StorageMode};
use serde_json::json;

let coll = VectorCollection::create(
    "./data/docs".into(),
    "docs",
    768,
    DistanceMetric::Cosine,
    StorageMode::Full,
)?;

coll.upsert(vec![
    Point::new(1, vec![0.1; 768], Some(json!({"title": "Hello"}))),
])?;

let results = coll.search(&vec![0.1; 768], 10)?;

Implementations

impl VectorCollection

pub fn diagnostics(&self) -> CollectionDiagnostics

Returns diagnostic information about this collection.

impl VectorCollection

pub fn guard_rails(&self) -> &Arc<GuardRails>

Returns a reference to the collection’s guard rails.

pub fn name(&self) -> String

Returns the collection name.

pub fn dimension(&self) -> usize

Returns the vector dimension.

pub fn metric(&self) -> DistanceMetric

Returns the distance metric.

pub fn storage_mode(&self) -> StorageMode

Returns the storage mode.

pub fn len(&self) -> usize

Returns the number of points in the collection.

pub fn is_empty(&self) -> bool

Returns true if the collection is empty.

pub fn all_ids(&self) -> Vec<u64>

Returns all point IDs.

pub fn scroll_batch( &self, cursor: Option<u64>, batch_size: usize, filter: Option<&Filter>, ) -> Result<ScrollBatch>

Returns the next batch of points for scroll iteration.

Delegates to the inner collection’s scroll implementation; see also the parallel scroll_batch on crate::GraphCollection and crate::MetadataCollection.

Errors

Returns an error if batch_size is 0.

pub fn config(&self) -> CollectionConfig

Returns the current collection config.

pub fn rebuild_index(&self) -> Result<usize>

Rebuilds the HNSW index of this collection from the vector storage, reclaiming memory occupied by tombstoned entries. Returns the number of entries compacted.

Used by the server admin endpoint POST /collections/{name}/index/rebuild (finding F-21).

Errors

Returns an error if the vacuum fails (for instance, when vector storage is disabled on the HNSW index).

pub fn compact_storage(&self) -> Result<usize>

Compacts the underlying vector storage, rewriting active vectors into a contiguous layout and reclaiming disk space occupied by deleted entries.

Returns the number of bytes reclaimed.

Used by the server admin endpoint POST /collections/{name}/compact.

Errors

Returns an error if the compaction I/O fails (e.g. disk full, file lock contention).

pub fn apply_advanced_config( &self, pq_rescore_oversampling: Option<Option<u32>>, deferred_indexing: Option<Option<DeferredIndexerConfig>>, async_index_builder: Option<Option<AsyncIndexBuilderConfig>>, ) -> Result<()>

Applies post-creation overrides to the advanced configuration fields (pq_rescore_oversampling, deferred_indexing, async_index_builder) and persists the updated config.json.

Each parameter is wrapped in an outer Option that expresses “leave unchanged” (None) versus “set to this value” (Some(_)). Passing Some(None) explicitly clears the inner field. A local clippy allow is applied because the three-state semantics are the intended contract here.

Errors

Returns an error if the updated config cannot be written to disk.

pub fn get_stats(&self) -> CollectionStats

Returns CBO statistics.

pub fn is_metadata_only(&self) -> bool

Returns true if the collection is a metadata-only collection.

pub fn analyze(&self) -> Result<CollectionStats>

Analyzes the collection and returns fresh statistics.

Errors

• Returns an error if statistics computation fails.

pub fn has_secondary_index(&self, field: &str) -> bool

Returns true if a secondary index exists on field.

pub fn drop_secondary_index(&self, field_name: &str) -> bool

Drops a secondary index on field_name. Returns true if the index existed.

pub fn has_property_index(&self, label: &str, property: &str) -> bool

Returns true if a property index exists.

pub fn has_range_index(&self, label: &str, property: &str) -> bool

Returns true if a range index exists.

pub fn list_indexes(&self) -> Vec<IndexInfo>

Lists all index definitions on this collection.

pub fn indexes_memory_usage(&self) -> usize

Returns total memory usage of all indexes in bytes.

pub fn attach_auto_reindex(&self, manager: Arc<AutoReindexManager>)

Attaches an AutoReindexManager to this collection as a runtime-only hook.

The attachment is not persisted to config.json — callers must re-attach after every Database::open. This intentional design avoids the Duration serde round-trip and keeps the collection schema version stable.

Once attached, the manager is consulted by the bulk upsert hot path after every successful upsert_bulk call. When the manager reports that index parameters have diverged from the optimal configuration for the current dataset size, a tracing::info! event is emitted. Automatic index reconstruction is NOT performed — that decision is left to the caller.

External consumers can register their own reindex pipeline via the manager’s event callback (crate::collection::auto_reindex::AutoReindexManager::on_event) or poll the divergence state via Self::check_auto_reindex_divergence.

Example

let manager = Arc::new(AutoReindexManager::new(AutoReindexConfig::default()));
coll.attach_auto_reindex(manager);

pub fn detach_auto_reindex(&self) -> Option<Arc<AutoReindexManager>>

Detaches the currently attached auto-reindex manager, returning it so callers can drop or reuse it. Returns None when no manager was attached.

pub fn auto_reindex_manager(&self) -> Option<Arc<AutoReindexManager>>

Returns a clone of the currently attached auto-reindex manager, or None if none is attached.

pub fn check_auto_reindex_divergence(&self) -> Option<DivergenceCheck>

Returns a DivergenceCheck computed from the attached manager against the collection’s current state, or None when no manager is attached.

Read-only — does not mutate the manager state.

impl VectorCollection

pub fn upsert_bulk(&self, points: &[Point]) -> Result<usize>

Bulk insert optimized for high-throughput import.

Errors

Returns an error if any point has a mismatched dimension.

pub fn upsert_bulk_from_raw( &self, vectors: &[f32], ids: &[u64], dimension: usize, payloads: Option<&[Option<Value>]>, ) -> Result<usize>

Bulk insert from contiguous flat slices (zero-copy from numpy / FFI).

Accepts a flat f32 slice of shape (n, dimension) in row-major order plus a matching u64 ID slice of length n. Avoids per-row Vec<f32> allocation, saving ~293 MB for 100K vectors at 768D.

Errors

• Returns crate::error::Error::InvalidVector if vectors.len() != ids.len() * dimension.
• Returns crate::error::Error::DimensionMismatch if dimension mismatches the collection.

pub fn upsert(&self, points: impl IntoIterator<Item = Point>) -> Result<()>

Inserts or updates points in the collection.

Errors

• Returns an error if any point’s dimension does not match the collection.
• Returns an error if storage operations fail.

Examples

coll.upsert(vec![
    Point::new(1, vec![0.1; 128], Some(json!({"title": "Hello"}))),
    Point::new(2, vec![0.2; 128], None),
])?;

pub fn get(&self, ids: &[u64]) -> Vec<Option<Point>>

Retrieves points by IDs, returning None for missing entries.

Examples

let points = coll.get(&[1, 2, 3]);
for (id, maybe_point) in [1, 2, 3].iter().zip(&points) {
    if let Some(p) = maybe_point {
        println!("Found point {id} with payload {:?}", p.payload);
    }
}

pub fn delete(&self, ids: &[u64]) -> Result<()>

Deletes points by IDs.

Missing IDs are silently ignored.

Errors

• Returns an error if storage operations fail.

Examples

coll.delete(&[1, 2, 3])?;

pub fn upsert_metadata( &self, points: impl IntoIterator<Item = Point>, ) -> Result<()>

Inserts or updates metadata-only points (no vectors).

Errors

• Returns an error if storage operations fail.

pub fn create_index(&self, field: &str) -> Result<()>

Creates a secondary metadata index on a payload field.

Errors

• Returns an error if the index already exists or storage fails.

pub fn create_property_index(&self, label: &str, property: &str) -> Result<()>

Creates a property index for O(1) equality lookups.

Errors

• Returns an error if the index already exists or storage fails.

pub fn create_range_index(&self, label: &str, property: &str) -> Result<()>

Creates a range index for O(log n) range queries.

Errors

• Returns an error if the index already exists or storage fails.

pub fn drop_index(&self, label: &str, property: &str) -> Result<bool>

Drops an index, returning true if an index was removed.

Errors

• Returns an error if the drop operation fails.

pub fn add_edge(&self, edge: GraphEdge) -> Result<()>

Adds a graph edge to the collection.

Errors

Returns an error if the edge cannot be stored.

impl VectorCollection

pub fn create( path: PathBuf, _name: &str, dimension: usize, metric: DistanceMetric, storage_mode: StorageMode, ) -> Result<Self>

Creates a new VectorCollection at the given path.

Errors

Returns an error if the directory cannot be created or storage fails.

pub fn create_with_hnsw( path: PathBuf, _name: &str, dimension: usize, metric: DistanceMetric, storage_mode: StorageMode, m: Option<usize>, ef_construction: Option<usize>, ) -> Result<Self>

Creates a new VectorCollection with custom HNSW parameters.

When m or ef_construction are Some, those values override the auto-tuned defaults. When both are None, this is equivalent to VectorCollection::create.

Shortcut for VectorCollection::create_with_params that only overrides max_connections and ef_construction; every other HNSW field stays at the dimension-based auto-tuned default, and pq_rescore_oversampling uses the engine default of Some(4).

Errors

Returns an error if the directory cannot be created or storage fails.

pub fn create_with_params( path: PathBuf, dimension: usize, metric: DistanceMetric, storage_mode: StorageMode, hnsw_params: HnswParams, pq_rescore_oversampling: Option<u32>, ) -> Result<Self>

Creates a new VectorCollection with a fully specified HnswParams and an explicit pq_rescore_oversampling override.

This is the most expressive constructor exposed by VectorCollection: callers pass the full params object directly, including alpha (VAMANA neighbour diversification), max_elements (initial HNSW capacity), and any future field added to HnswParams, without going through the (m, ef_construction) shortcut. Passing pq_rescore_oversampling = None keeps the persisted config in “no explicit override” mode so later migrations can recompute the factor from dataset shape.

Errors

Returns an error if the directory cannot be created or storage fails.

pub fn open(path: PathBuf) -> Result<Self>

Opens an existing VectorCollection from disk.

Errors

Returns an error if the config file cannot be read or storage cannot be opened.

pub fn create_with_async_builder( path: PathBuf, dimension: usize, metric: DistanceMetric, async_builder_config: AsyncIndexBuilderConfig, ) -> Result<Self>

Creates a new VectorCollection with an async index builder configuration.

Errors

Returns an error if the directory cannot be created or the config cannot be saved.

pub fn flush(&self) -> Result<()>

Flushes all engines to disk and saves the config.

Issue #423: This fast-path flush skips vectors.idx serialization. The WAL provides crash recovery for the vector index.

Errors

Returns an error if any flush operation fails.

pub fn flush_full(&self) -> Result<()>

Full durability flush including vectors.idx serialization.

Issue #423: Use on graceful shutdown to avoid a full WAL replay on the next startup.

Errors

Returns an error if any flush operation fails.

impl VectorCollection

pub fn search(&self, query: &[f32], k: usize) -> Result<Vec<SearchResult>>

Performs kNN vector search using the HNSW index.

Returns the k nearest neighbors ordered by ascending distance.

Errors

• Returns an error if the query dimension does not match the collection.
• Returns an error if the HNSW index is not initialized.

Examples

let results = coll.search(&vec![0.1; 128], 10)?;
for r in &results {
    println!("id={} score={}", r.point.id, r.score);
}

pub fn text_search(&self, query: &str, k: usize) -> Result<Vec<SearchResult>>

Performs full-text BM25 search over indexed payload fields.

Returns up to k results ranked by BM25 relevance score.

Errors

• Returns an error if storage retrieval fails.

Examples

let results = coll.text_search("machine learning", 5)?;

pub fn search_with_ef( &self, query: &[f32], k: usize, ef_search: usize, ) -> Result<Vec<SearchResult>>

Performs kNN search with an explicit ef_search override.

Higher ef_search values improve recall at the cost of latency.

Errors

• Returns an error if the query dimension does not match the collection.

pub fn search_with_quality( &self, query: &[f32], k: usize, quality: SearchQuality, ) -> Result<Vec<SearchResult>>

Performs kNN search with a specific crate::SearchQuality profile.

Use this instead of Self::search_with_ef when you want named quality modes like crate::SearchQuality::AutoTune that compute ef dynamically.

Errors

• Returns an error if the query dimension does not match the collection.

pub fn search_with_filter( &self, query: &[f32], k: usize, filter: &Filter, ) -> Result<Vec<SearchResult>>

Performs kNN search with a metadata filter applied post-retrieval.

Errors

• Returns an error if the query dimension does not match the collection.
• Returns an error if the filter references an unsupported field type.

pub fn search_ids(&self, query: &[f32], k: usize) -> Result<Vec<ScoredResult>>

Returns crate::ScoredResult pairs without payload hydration.

Faster than search when only IDs and scores are needed.

Errors

• Returns an error if the query dimension does not match the collection.

pub fn text_search_with_filter( &self, query: &str, k: usize, filter: &Filter, ) -> Result<Vec<SearchResult>>

Full-text search with metadata filter.

Errors

Returns an error if storage retrieval fails.

pub fn hybrid_search( &self, vector: &[f32], text: &str, k: usize, alpha: Option<f32>, ) -> Result<Vec<SearchResult>>

Performs hybrid search combining vector kNN and BM25 full-text via RRF fusion.

When alpha is None, a default blending factor is used. Values closer to 1.0 weight vector results more; values closer to 0.0 weight text.

Errors

• Returns an error if the query dimension does not match the collection.
• Returns an error if text indexing or storage retrieval fails.

Examples

let results = coll.hybrid_search(&vec![0.1; 128], "machine learning", 10, Some(0.7))?;

pub fn hybrid_search_with_filter( &self, vector: &[f32], text: &str, k: usize, alpha: Option<f32>, filter: &Filter, ) -> Result<Vec<SearchResult>>

Performs hybrid search (vector + BM25) with a metadata filter.

Errors

• Returns an error if the query dimension does not match the collection.
• Returns an error if text indexing, storage, or filtering fails.

pub fn search_batch_with_filters( &self, queries: &[&[f32]], k: usize, filters: &[Option<Filter>], ) -> Result<Vec<Vec<SearchResult>>>

Performs batch kNN search with per-query metadata filters.

Each query in queries is paired with the filter at the same index in filters. Pass None for queries that should not be filtered.

Errors

• Returns an error if any query dimension does not match the collection.
• Returns an error if queries and filters have different lengths.

Examples

let q1 = vec![0.1; 128];
let q2 = vec![0.2; 128];
let results = coll.search_batch_with_filters(
    &[q1.as_slice(), q2.as_slice()],
    10,
    &[None, None],
)?;
assert_eq!(results.len(), 2);

pub fn search_batch_parallel( &self, queries: &[&[f32]], k: usize, ) -> Result<Vec<Vec<SearchResult>>>

Performs batch kNN search without filters, optimized for throughput.

Uses rayon-parallelized HNSW search and result resolution for maximum queries-per-second. Prefer this over calling search in a loop.

Errors

• Returns an error if any query dimension does not match the collection.

Examples

let q1 = vec![0.1; 128];
let q2 = vec![0.2; 128];
let results = coll.search_batch_parallel(&[q1.as_slice(), q2.as_slice()], 10)?;
assert_eq!(results.len(), 2);

pub fn multi_query_search( &self, queries: &[&[f32]], k: usize, strategy: FusionStrategy, filter: Option<&Filter>, ) -> Result<Vec<SearchResult>>

Performs multi-query search fusing results from multiple query vectors.

Errors

• Returns an error if any query dimension does not match the collection.
• Returns an error if the fusion strategy fails.

pub fn multi_query_search_ids( &self, queries: &[&[f32]], k: usize, strategy: FusionStrategy, ) -> Result<Vec<(u64, f32)>>

Performs multi-query search returning only IDs and fused scores.

Errors

• Returns an error if any query dimension does not match the collection.
• Returns an error if the fusion strategy fails.

pub fn sparse_search( &self, query: &SparseVector, k: usize, index_name: &str, ) -> Result<Vec<SearchResult>>

Performs sparse-only search on the named index.

Errors

Returns an error if the named sparse index does not exist.

pub fn hybrid_sparse_search( &self, dense_vector: &[f32], sparse_query: &SparseVector, k: usize, index_name: &str, strategy: &FusionStrategy, ) -> Result<Vec<SearchResult>>

Performs hybrid dense+sparse search with RRF fusion.

Errors

Returns an error if dense or sparse search fails, or fusion errors.

pub fn execute_match( &self, match_clause: &MatchClause, params: &HashMap<String, Value>, ) -> Result<Vec<MatchResult>>

Executes a graph MATCH query against the collection’s edge store.

Errors

• Returns an error if the match clause references an invalid label or property.
• Returns an error if the edge store is not initialized.

pub fn execute_match_with_similarity( &self, match_clause: &MatchClause, query_vector: &[f32], threshold: f32, params: &HashMap<String, Value>, ) -> Result<Vec<MatchResult>>

Executes a MATCH query with vector similarity filtering.

Errors

• Returns an error if the match clause is invalid or the query dimension mismatches.

pub fn execute_aggregate( &self, query: &Query, params: &HashMap<String, Value>, ) -> Result<Value>

Executes an aggregation query (GROUP BY / COUNT / SUM / AVG / MIN / MAX).

Errors

• Returns an error if the query is invalid or aggregation computation fails.

pub fn execute_query( &self, query: &Query, params: &HashMap<String, Value>, ) -> Result<Vec<SearchResult>>

Executes a parsed VelesQL query.

Errors

• Returns an error if the query references missing fields or execution fails.

pub fn explain_analyze_query( &self, query: &Query, params: &HashMap<String, Value>, ) -> Result<ExplainOutput>

Executes a query with instrumentation and returns plan + actual stats.

Delegates to crate::Database::explain_analyze_query.

Errors

Returns an error if the query is invalid or execution fails.

pub fn stream_insert(&self, point: Point) -> Result<(), BackpressureError>

Sends a point into the streaming ingestion channel.

Returns Ok(()) on success (202 semantics). Returns BackpressureError::BufferFull when the channel is at capacity, or BackpressureError::NotConfigured if streaming is not active.

Errors

Returns BackpressureError on buffer-full or not-configured.

pub fn stream_insert_batch( &self, points: Vec<Point>, ) -> Result<usize, BackpressureError>

Sends a batch of points into the streaming ingestion channel.

Acquires the ingester lock once for the entire batch, eliminating per-point lock overhead. Returns the number of points successfully queued. Companion to Self::stream_insert for single-point sends.

Errors

Returns BackpressureError on buffer-full, drain-dead, or not-configured.

pub fn push_to_delta_if_active(&self, entries: &[(u64, Vec<f32>)])

Pushes (id, vector) entries into the delta buffer if it is active.

No-op when the delta buffer is inactive. This is the public interface used by streaming upsert handlers (e.g., NDJSON stream endpoint) to keep the delta buffer in sync after a successful upsert_bulk call.

pub fn is_delta_active(&self) -> bool

Returns true if the delta buffer is currently active (HNSW rebuild in progress). External callers can use this to decide whether to snapshot entries for delta before a upsert_bulk call.

pub fn enable_streaming(&self, config: StreamingConfig)

Enables streaming ingestion on this collection.

Creates a StreamIngester with the given config and stores it internally. Points can then be submitted via stream_insert or stream_insert_batch.

Calling this when streaming is already active replaces the existing ingester (the old drain task is aborted via Drop).

pub fn execute_query_str( &self, sql: &str, params: &HashMap<String, Value>, ) -> Result<Vec<SearchResult>>

Executes a raw VelesQL string, parsing it before execution.

Errors

• Returns an error if the SQL string cannot be parsed.
• Returns an error if query execution fails.

pub fn reorder_for_locality(&self) -> Result<()>

Reorders HNSW graph nodes in BFS traversal order for improved cache locality.

After bulk insertion, nodes are stored in insertion order. Calling this method once after loading vectors reorders both the vector buffer and all adjacency lists so nodes traversed together during search are close in memory, reducing L2/L3 cache misses by 15–30% on collections with ≥ 1 000 vectors (issue #377).

Also builds a PDX block-columnar layout for SIMD-parallel distance computation when the columnar search path is enabled.

When to call

After Self::upsert bulk-loading for a new collection, before the collection is opened for queries. No-op for collections with fewer than 1 000 vectors.

Errors

Returns an error if vector storage reordering fails.

Trait Implementations

impl Clone for VectorCollection

fn clone(&self) -> VectorCollection

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.