Struct SparseVectorConfig 

pub struct SparseVectorConfig {

    pub format: SparseFormat,
    pub index_size: IndexSize,
    pub weight_quantization: WeightQuantization,
    pub weight_threshold: f32,
    pub doc_mass: Option<f32>,
    pub block_size: usize,
    pub bmp_block_size: u32,
    pub bmp_grid_bits: u8,
    pub bmp_superblock_size: u32,
    pub pruning: Option<f32>,
    pub query_config: Option<SparseQueryConfig>,
    pub dims: Option<u32>,
    pub max_weight: Option<f32>,
    pub min_terms: usize,
}

Configuration for sparse vector storage

Research-validated optimizations for learned sparse retrieval (SPLADE, uniCOIL, etc.):

• Weight threshold (0.01-0.05): Removes ~30-50% of postings with minimal nDCG impact
• Posting list pruning (0.1): Keeps top 10% per dimension, 50-70% index reduction, <1% nDCG loss
• Query pruning (top 10-20 dims): 30-50% latency reduction, <2% nDCG loss
• UInt8 quantization: 4x compression, 1-2% nDCG loss (optimal trade-off)

Fields

format: SparseFormat

Index format: MaxScore (DAAT) or BMP (BAAT)

index_size: IndexSize

Size of dimension/term indices

weight_quantization: WeightQuantization

Quantization for weights (see WeightQuantization docs for trade-offs)

weight_threshold: f32

Minimum weight threshold - weights below this value are not indexed

Research recommendation (Guo et al., 2022; SPLADE v2):

• 0.01-0.05 for SPLADE models removes ~30-50% of postings
• Minimal impact on nDCG@10 (<1% loss)
• Major reduction in index size and query latency

doc_mass: Option<f32>

Document-side mass cropping: keep the top-|weight| entries covering this fraction of a sparse vector’s total |weight| mass; the excessive tail is dropped at indexing time.

SPLADE-style vectors concentrate importance in a few head terms; the long tail inflates the index and query cost with little relevance signal. 0.9-0.95 typically drops 20-40% of postings with <1% nDCG loss.

• None or >= 1.0 = keep all entries (default)
• Applied after weight_threshold; vectors with <= min_terms entries are never cropped.

block_size: usize

Block size for posting lists (must be power of 2, default 128 for SIMD) Larger blocks = better compression, smaller blocks = faster seeks. Used by MaxScore format only.

bmp_block_size: u32

BMP block size: number of consecutive doc_ids per block (must be power of 2, max 256). Only used when format = Bmp. Uniform across every segment of the field — set per field in SDL (bmp_block_size: N). Smaller = better pruning granularity; larger = smaller grid — the dense 4-bit grid is dims × num_blocks / 2 bytes, so grid memory scales as 1/block_size. Default 64; large corpora (10M+ docs at 100k-dim vocabularies) should set 256 to bound grid memory (docs/bmp-grid-compression.md).

bmp_grid_bits: u8

Bits per BMP block-grid cell: 4 (default) or 2. The grid is dims × num_blocks × bits/8 bytes, so 2 halves grid memory; measured pruning cost is ~free (+0.4-2.2% blocks scored — the exact u8 sb_grid prunes first and shields the block grid; docs/bmp-grid-compression.md). Grid bounds are ceil-quantized, so exact top-k results are unchanged at any width. Uniform per field across all segments — set in SDL (bmp_grid_bits: 2) at index creation.

bmp_superblock_size: u32

BMP superblock size: number of consecutive blocks grouped for hierarchical pruning (Carlson et al., SIGIR 2025). Must be power of 2. Default 64. Set to 0 to disable superblock pruning (flat BMP scoring). Only used when format = Bmp.

pruning: Option<f32>

Static pruning: fraction of postings to keep per inverted list (SEISMIC-style) Lists are sorted by weight descending and truncated to top fraction.

Research recommendation (SPLADE v2, Formal et al., 2021):

• None = keep all postings (default, exact)
• Some(0.1) = keep top 10% of postings per dimension
  • 50-70% index size reduction
  • <1% nDCG@10 loss
  • Exploits “concentration of importance” in learned representations

Applied only during initial segment build, not during merge.

query_config: Option<SparseQueryConfig>

Query-time configuration (tokenizer, weighting)

dims: Option<u32>

Fixed vocabulary size (number of dimensions) for BMP format.

When set, all BMP segments use the same grid dimensions (rows = dims), enabling zero-copy block-copy merge. The grid is indexed by dim_id directly (no dim_ids Section C needed).

Required for BMP format. Typical values:

• SPLADE/BERT: 30522 or 105879 (WordPiece / Unigram vocabulary)
• uniCOIL: 30522
• Custom models: set to vocabulary size

If None, BMP builder derives dims from observed data (V10 behavior).

max_weight: Option<f32>

Fixed max weight scale for BMP format.

When set, all BMP segments use the same quantization scale (max_weight_scale = max_weight), eliminating rescaling during merge.

For SPLADE models: 5.0 (covers typical weight range 0-5). If None, BMP builder derives scale from data (V10 behavior).

min_terms: usize

Minimum number of postings in a dimension before pruning and weight_threshold filtering are applied. Protects dimensions with very few postings from losing most of their signal.

Default: 4. Set to 0 to always apply pruning/filtering.

Implementations

impl SparseVectorConfig

pub fn splade() -> Self

SPLADE-optimized config with research-validated defaults

Optimized for SPLADE, uniCOIL, and similar learned sparse retrieval models. Based on research findings from:

• Pati (2025): UInt8 quantization = 4x compression, 1-2% nDCG loss
• Formal et al. (2021): SPLADE v2 posting list pruning
• Qiao et al. (2023): Query dimension pruning and approximate search
• Guo et al. (2022): Weight thresholding for efficiency

Expected performance vs. full precision baseline:

• Index size: ~15-25% of original (combined effect of all optimizations)
• Query latency: 40-60% faster
• Effectiveness: 2-4% nDCG@10 loss (typically acceptable for production)

Vocabulary: ~30K dimensions (fits in u16)

pub fn splade_bmp() -> Self

SPLADE-optimized config with BMP (Block-Max Pruning) format

Same optimization settings as splade() but uses the BMP block-at-a-time format (Mallia, Suel & Tonellotto, SIGIR 2024) instead of MaxScore. BMP divides the document space into fixed-size blocks and processes them in decreasing upper-bound order, enabling aggressive early termination.

pub fn compact() -> Self

Compact config: Maximum compression (experimental)

Uses aggressive UInt4 quantization for smallest possible index size. Expected trade-offs:

• Index size: ~10-15% of Float32 baseline
• Effectiveness: ~3-5% nDCG@10 loss

Recommended for: Memory-constrained environments, cache-heavy workloads

pub fn full_precision() -> Self

Full precision config: No compression, baseline effectiveness

Use for: Research baselines, when effectiveness is critical

pub fn conservative() -> Self

Conservative config: Mild optimizations, minimal effectiveness loss

Balances compression and effectiveness with conservative defaults. Expected trade-offs:

• Index size: ~40-50% of Float32 baseline
• Query latency: ~20-30% faster
• Effectiveness: <1% nDCG@10 loss

Recommended for: Production deployments prioritizing effectiveness

pub fn with_weight_threshold(self, threshold: f32) -> Self

Set weight threshold (builder pattern)

pub fn with_doc_mass(self, fraction: f32) -> Self

Set document-side mass cropping fraction (builder pattern) e.g., 0.9 = keep top-weight entries covering 90% of each vector’s mass

pub fn with_pruning(self, fraction: f32) -> Self

Set posting list pruning fraction (builder pattern) e.g., 0.1 = keep top 10% of postings per dimension

pub fn bytes_per_entry(&self) -> f32

Bytes per entry (index + weight)

pub fn to_byte(&self) -> u8

Serialize config to a single byte.

Layout: bits 7-4 = IndexSize, bit 3 = format (0=MaxScore, 1=BMP), bits 2-0 = WeightQuantization

pub fn from_byte(b: u8) -> Option<Self>

Deserialize config from a single byte.

Note: weight_threshold, block_size, bmp_block_size, and query_config are not serialized in the byte — they come from the schema.

pub fn with_block_size(self, size: usize) -> Self

Set block size (builder pattern) Must be power of 2, recommended: 64, 128, 256

pub fn with_query_config(self, config: SparseQueryConfig) -> Self

Set query configuration (builder pattern)

Trait Implementations

impl Clone for SparseVectorConfig

fn clone(&self) -> SparseVectorConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for SparseVectorConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for SparseVectorConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for SparseVectorConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for SparseVectorConfig

fn eq(&self, other: &SparseVectorConfig) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for SparseVectorConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for SparseVectorConfig