Module segment 

Immutable index segments and their lifecycle building blocks.

Builders write a complete segment, readers expose its text, stored-field, and vector data, and native mergers combine committed segments. Publication and deletion are coordinated at the index layer by crate::merge::SegmentManager.

Modules

chunk_map

Virtual-id maps of chunked text fields (seg_<id>.chunks).

pin

Hot-metadata pinning: budgeted residency for per-query-mandatory structures (a meta/data residency split).

Structs

AnnHealth

Cheap structural health of one ANN payload, computed from the in-memory run directory in O(runs) — payload bytes are never touched.

AsyncStoreReader

Async document store reader - loads blocks on demand

BmpDimStats

Summary statistics for per-dimension postings in a BMP sparse index.

BpBudget

CPU/depth budget for a BP pass. BP is an anytime algorithm: stopping at any depth or deadline still yields a valid permutation, and because the output layout becomes the next pass’s input order, repeated budgeted passes warm-start and deepen (top levels converge in ~0 swaps, the budget flows to deeper levels).

DensePlanCache

Per-query dense plan caches, shared by every segment scorer the query spawns. Both members are query-global: the IVF-TQ probe route and its LUTs depend only on the query and index-level artifacts, and the TQ LUTs depend only on the query and the schema dimension.

EagerParallelStoreWriter

Parallel document store writer - compresses blocks immediately when queued

FieldStats

Field statistics for BM25F scoring

FlatVectorData

Flat vector binary format helpers for writing.

LazyFlatVectorData

Lazy flat vector data — zero-copy doc_id index, vectors via range reads.

MemoryBreakdown

Detailed memory breakdown by component

MergeStats

Statistics for merge operations

PublishedIndexGeneration

Immutable schema and trained-vector artifacts published with one segment generation. Readers retain this object with their segment references, so a vector-index ALTER cannot pair old segments with new routing parameters.

RawStoreBlock

Raw block info for store merging (without decompression)

ScannTrainedArtifactBytes

Mmap/Arc-backed bytes for one validated global ScaNN generation. The full artifact is fingerprinted exactly once at open; query/merge compatibility reads the cached identity/config instead of rehashing the centroid plane.

SegmentBuilder

Segment builder with optimized memory usage

SegmentBuilderConfig

Configuration for segment builder

SegmentBuilderStats

Statistics about segment builder state

SegmentFiles

Paths for segment files

SegmentId

Unique segment identifier (UUID7-like: 48-bit timestamp + 80-bit random)

SegmentMerger

Segment merger - merges multiple segments into one

SegmentMeta

Segment metadata

SegmentReader

Async segment reader with lazy loading

SegmentSnapshot

RAII guard that holds references to a snapshot of segments. When dropped, releases all segment references and triggers deferred deletion.

SegmentTracker

Tracks segment references and pending deletions

SparseIndex

Sparse vector index for a field: lazy block loading via mmap.

StoreMerger

Store merger - concatenates compressed blocks from multiple stores without recompression

TrainedVectorStructures

Trained vector index structures for rebuilding segments with ANN indexes

VectorSearchResult

Vector search result with ordinal tracking for multi-value fields

Enums

VectorIndex

Production ANN payloads for float IVF-PQ and packed-binary IVF.

Constants

DEFAULT_DICT_SIZE

Default dictionary size (4KB is a good balance)

STORE_BLOCK_SIZE

Block size for document store (16KB). Smaller blocks reduce read amplification for single-doc fetches at the cost of slightly worse compression ratio. Zstd dictionary training recovers most of the compression loss.

Functions

delete_segment

Delete segment files from directory (all deletions run concurrently).

dequantize_raw

Dequantize raw bytes to f32 based on storage quantization.

deserialize_document

Deserialize all fields from document bytes.

deserialize_document_fields

Deserialize only specific fields from document bytes.

serialize_document

Binary document format: num_fields: u32 per field: field_id: u16, type_tag: u8, value data 0=Text: len:u32 + utf8 1=U64: u64 LE 2=I64: i64 LE 3=F64: f64 LE 4=Bytes: len:u32 + raw 5=SparseVector: count:u32 + count*(u32+f32) 6=DenseVector: count:u32 + count*f32 7=Json: len:u32 + json utf8

serialize_document_into

Serialize a document into a reusable buffer (clears it first). Avoids per-document allocation when called in a loop.