Module segments 

Segments: an index is a list of immutable files plus what is still in RAM.

Until 3.0.5 a sparse index was one sparse.mmap, rewritten whole at every commit — so inserting one vector cost what inserting the whole index cost (320 ms at 200 000 vectors, growing with the file: see tests/bench_commit_cost.rs). A commit now writes one segment holding the vectors added since the last one, and meta.json lists what is active. The cost of a commit is the cost of the delta.

A segment is a version 3 file: its dimension table is keyed by global token id and sorted (see crate::mmap_index). That is what lets two segments — and therefore two indexes — be merged by walking their tables together, without remapping anything.

Deletions are tombstones. Removing a document marks its id in the segments that hold it; a segment written afterwards is not concerned, which is what makes an update (delete, then insert) land the right way round. A merge applies them and clears the lists.

Knowing which segment holds an id is what seg_<id>.ids is for: the segment’s record ids, sorted, eight bytes each, read only when something is deleted or updated. It replaces the far larger sparse_vectors.bin (whole vectors, kept only to know which dimensions to touch on a deletion), and it hands a merge its id list for nothing.

Structs

IndexMeta

The index’s manifest: which segments are active, and what is deleted.

Segment

An open segment: its manifest entry and its mapping.

SegmentMeta

One segment of an index.

Constants

META_FILE

meta.json — what this index is made of.

META_VERSION

What this build writes; a newer one is refused rather than misread.

Functions

encode_ids

Serialize sorted ids for ids_file.

ids_file

seg_<id>.ids — the segment’s record ids, sorted, little-endian.

merge_segments

Merge segments into one, applying their tombstones.

new_segment_id

An id for a new segment, from the process, a counter and the clock — unique within an index without needing to look at what is already there.

segment_file

seg_<id>.mmap.