Skip to main contentModule segment
Source - AsyncSegmentReader
- Async segment reader with lazy loading
- AsyncStoreReader
- Async document store reader - loads blocks on demand
- EagerParallelStoreWriter
- Parallel document store writer - compresses blocks immediately when queued
- FieldStats
- Field statistics for BM25F scoring
- FlatVectorData
- Flat vector data for brute-force search (accumulating state)
- IVFRaBitQIndexData
- IVF-RaBitQ index data with embedded centroids and codebook
- MemoryBreakdown
- Detailed memory breakdown by component
- MergeStats
- Statistics for merge operations
- RawStoreBlock
- Raw block info for store merging (without decompression)
- ScaNNIndexData
- ScaNN index data with embedded centroids and codebook
- SegmentBuilder
- Segment builder with optimized memory usage
- SegmentBuilderConfig
- Configuration for segment builder
- SegmentBuilderStats
- Statistics about segment builder state
- SegmentFiles
- Paths for segment files
- SegmentId
- Unique segment identifier (UUID7-like: 48-bit timestamp + 80-bit random)
- SegmentMerger
- Segment merger - merges multiple segments into one
- SegmentMeta
- Segment metadata
- SegmentSnapshot
- RAII guard that holds references to a snapshot of segments
When dropped, releases all segment references
- SegmentTracker
- Tracks segment references and pending deletions
- SparseIndex
- Sparse vector index for a field: lazy block loading via mmap
- StoreMerger
- Store merger - concatenates compressed blocks from multiple stores without recompression
- TrainedVectorStructures
- Trained vector index structures for rebuilding segments with ANN indexes
- VectorSearchResult
- Vector search result with ordinal tracking for multi-value fields
- VectorIndex
- Vector index type - Flat, RaBitQ, IVF-RaBitQ, or ScaNN (IVF-PQ)
- DEFAULT_DICT_SIZE
- Default dictionary size (64KB is a good balance)
- STORE_BLOCK_SIZE
- Block size for document store (256KB for better compression)
Larger blocks = better compression ratio but more memory per block load
- delete_segment
- Delete segment files from directory
- deserialize_document
- serialize_document
- Binary document format:
num_fields: u16
per field: field_id: u16, type_tag: u8, value data
0=Text: len:u32 + utf8
1=U64: u64 LE
2=I64: i64 LE
3=F64: f64 LE
4=Bytes: len:u32 + raw
5=SparseVector: count:u32 + count*(u32+f32)
6=DenseVector: count:u32 + count*f32
7=Json: len:u32 + json utf8
- SegmentReader
- Alias for AsyncSegmentReader