Skip to main contentModule segment
Source - AsyncSegmentReader
- Async segment reader with lazy loading
- AsyncStoreReader
- Async document store reader - loads blocks on demand
- EagerParallelStoreWriter
- Parallel document store writer - compresses blocks immediately when queued
- FieldStats
- Field statistics for BM25F scoring
- FlatVectorData
- Flat vector binary format helpers for writing.
- IVFRaBitQIndexData
- IVF-RaBitQ index data (codebook + cluster assignments)
- LazyFlatVectorData
- Lazy flat vector data — zero-copy doc_id index, vectors via range reads.
- MemoryBreakdown
- Detailed memory breakdown by component
- MergeStats
- Statistics for merge operations
- RawStoreBlock
- Raw block info for store merging (without decompression)
- ScaNNIndexData
- ScaNN index data (codebook + cluster assignments)
- SegmentBuilder
- Segment builder with optimized memory usage
- SegmentBuilderConfig
- Configuration for segment builder
- SegmentBuilderStats
- Statistics about segment builder state
- SegmentFiles
- Paths for segment files
- SegmentId
- Unique segment identifier (UUID7-like: 48-bit timestamp + 80-bit random)
- SegmentMerger
- Segment merger - merges multiple segments into one
- SegmentMeta
- Segment metadata
- SegmentSnapshot
- RAII guard that holds references to a snapshot of segments.
When dropped, releases all segment references and triggers deferred deletion.
- SegmentTracker
- Tracks segment references and pending deletions
- SparseIndex
- Sparse vector index for a field: lazy block loading via mmap.
- StoreMerger
- Store merger - concatenates compressed blocks from multiple stores without recompression
- TrainedVectorStructures
- Trained vector index structures for rebuilding segments with ANN indexes
- VectorSearchResult
- Vector search result with ordinal tracking for multi-value fields
- VectorIndex
- Vector index type - RaBitQ, IVF-RaBitQ, or ScaNN (IVF-PQ)
- DEFAULT_DICT_SIZE
- Default dictionary size (4KB is a good balance)
- STORE_BLOCK_SIZE
- Block size for document store (256KB for better compression)
Larger blocks = better compression ratio but more memory per block load
- delete_segment
- Delete segment files from directory (all deletions run concurrently).
- dequantize_raw
- Dequantize raw bytes to f32 based on storage quantization.
- deserialize_document
- Deserialize all fields from document bytes.
- deserialize_document_fields
- Deserialize only specific fields from document bytes.
- serialize_document
- Binary document format:
num_fields: u16
per field: field_id: u16, type_tag: u8, value data
0=Text: len:u32 + utf8
1=U64: u64 LE
2=I64: i64 LE
3=F64: f64 LE
4=Bytes: len:u32 + raw
5=SparseVector: count:u32 + count*(u32+f32)
6=DenseVector: count:u32 + count*f32
7=Json: len:u32 + json utf8
- serialize_document_into
- Serialize a document into a reusable buffer (clears it first).
Avoids per-document allocation when called in a loop.
- SegmentReader
- Alias for AsyncSegmentReader