sparse-vector 4.2.0

Sparse vector inverted index with WAND pruning, persisted through lucistore and sharded through luciole — a lucivy friend crate. Original code, design inspired by Qdrant's sparse index (see NOTICE).
Documentation

sparse-vector

Inverted index for sparse vectors (SPLADE, BM25-as-vectors, learned sparse embeddings) with WAND pruning. A friend crate of lucivy: it persists through lucistore (filesystem or any BlobStore) and shards behind luciole actors, on the same router as the full-text index.

Features

  • Dense remapping — token ids are remapped to dense dimensions; each dimension owns a posting list whose elements carry a suffix-maximum ceiling.
  • WAND search — windows of record ids are scored at once, ranges that cannot reach the current top-k are skipped. Exact results: pruning never changes the ranking, only the work.
  • mmap or RAM — an index is a flat mmap file plus small side files; search runs directly over the mapping.
  • Filtered search — search_filtered(query, limit, allowed_ids) seeks the allowed ids when they are few, falls back to a set filter otherwise.
  • Sharding — ShardedSparseHandle: N shards, round-robin or locality-aware routing (balance_weight), routed filtered search (only the shards holding allowed ids work), shard_for_node_id.
  • Storage backends — FsSparseStorage (directories) or BlobSparseStorage<S: BlobStore> (blobs are the source of truth, a local cache holds the mmaps). Bring your own store: SQL, S3, memory.

Quick start

use sparse_vector::handle::SparseHandle;
use sparse_vector::index::SparseVector;

let index = SparseHandle::create("/tmp/sparse_demo")?;
index.insert(1, &SparseVector::new(vec![3, 17, 42], vec![0.8, 0.2, 1.1]))?;
index.insert(2, &SparseVector::new(vec![17, 99], vec![0.5, 0.9]))?;
index.commit_inner()?;

let hits = index.search(&SparseVector::new(vec![17, 42], vec![1.0, 1.0]), 10);
// [(1, 1.3), (2, 0.5)] — (node_id, dot product), best first
# Ok::<(), String>(())

Sharded, over a blob store:

use std::path::Path;
use std::sync::Arc;
use lucistore::blob_store::MemBlobStore;
use sparse_vector::sharded::{ShardedSparseConfig, ShardedSparseHandle};

let store = Arc::new(MemBlobStore::new());
let index = ShardedSparseHandle::create_with_store(
    store, "vectors", Path::new("/tmp/sparse_cache"), &ShardedSparseConfig::new(4))?;
// insert / remove / commit / search / search_filtered / close / drop_index
# Ok::<(), String>(())

Search options

wand::SearchOptions controls the loop: pruning (on by default), window (ids scored per batch, 4096 by default). SearchOptions::exhaustive() disables pruning — useful as ground truth in tests.

Design

The design — dimension remapping, ceilings on posting lists, batch scoring of id windows, WAND-style pruning — is inspired by the sparse index of Qdrant. The code is original; see NOTICE.

License

MIT.