Struct SparseIndex 

pub struct SparseIndex {
    pub total_docs: u32,
    pub total_vectors: u32,
    /* private fields */
}

Sparse vector index for a field: lazy block loading via mmap.

V3 layout optimizations:

• SoA dimension table: binary search only touches dim_ids array (352KB for 88K dims, fits L2 cache) instead of 2.8MB AoS.
• OwnedBytes skip section: zero-copy mmap reference to the contiguous skip-entry region at the tail of the file. No heap allocation or parsing during segment load.
• Single tail read: only footer + TOC + skip section are read during load; block data at the front of the file is never touched.

Fields

total_docs: u32

Total document count in this segment (for IDF computation)

total_vectors: u32

Total sparse vectors in this segment (for multi-valued IDF)

Implementations

impl SparseIndex

pub fn new( handle: LazyFileHandle, dims: DimensionTable, skip_bytes: OwnedBytes, total_docs: u32, total_vectors: u32, ) -> Self

Create a new V3 sparse index with SoA dimension table and zero-copy skip section

pub async fn get_posting( &self, dim_id: u32, ) -> Result<Option<Arc<BlockSparsePostingList>>>

Get posting list for a dimension (loads all blocks via mmap)

pub fn get_skip_list(&self, dim_id: u32) -> Option<(Vec<SparseSkipEntry>, f32)>

Get skip list for a dimension (for block-max iteration without loading blocks). Returns owned Vec since entries are parsed from zero-copy mmap bytes.

pub async fn get_block( &self, dim_id: u32, block_idx: usize, ) -> Result<Option<SparseBlock>>

Load specific block for a dimension

pub fn has_dimension(&self, dim_id: u32) -> bool

Check if dimension exists

pub fn num_dimensions(&self) -> usize

Get the number of dimensions in the index

pub fn active_dimensions(&self) -> impl Iterator<Item = u32> + '_

Iterate over all active dimension IDs

pub fn doc_count(&self, dim_id: u32) -> u32

Get doc count for dimension (from SoA table, no I/O)

pub fn idf(&self, dim_id: u32) -> f32

Compute IDF using doc_count from SoA table

pub fn idf_weights(&self, dim_ids: &[u32]) -> Vec<f32>

Get IDF weights for multiple dimensions

pub fn estimated_memory_bytes(&self) -> usize

Estimated memory usage in bytes

• DimensionTable: SoA arrays (6 × ndims × 4 bytes)
• Skip section: OwnedBytes (Arc overhead only; data is mmap-backed)
• Handle: LazyFileHandle overhead

pub async fn read_dim_raw(&self, dim_id: u32) -> Result<Option<DimRawData>>

Get merge-level info for a dimension: skip entries, doc_count, global_max_weight, and raw block data bytes (single mmap read, zero deserialization).

Trait Implementations

impl Clone for SparseIndex

fn clone(&self) -> SparseIndex

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.