sparse_vector/lib.rs
1//! sparse-vector: an inverted index for sparse vectors with WAND pruning.
2//!
3//! Token ids are remapped to dense dimensions, each dimension holds a
4//! posting list whose elements carry a suffix-maximum ceiling, and a query
5//! is answered by the [`wand`] search loop: windows of record ids are
6//! scored at once and the ranges that cannot reach the top-k are skipped.
7//! Indexes persist as a flat mmap file plus bincode side files, through the
8//! filesystem or a lucistore `BlobStore`, and shard behind luciole actors.
9//!
10//! The design follows the sparse index of Qdrant (dimension remapping,
11//! ceilings on posting lists, batch scoring); the code is original.
12//!
13//! Lives in the lucivy workspace as a friend crate: it persists through
14//! `lucistore` (`BlobStore`) like the FTS index, and is meant to share its
15//! storage, sharding and sync machinery. rag3weaver drives it from Rust.
16
17pub mod blob_store;
18pub mod handle;
19pub mod index;
20pub mod mmap_index;
21pub mod sharded;
22pub mod wand;