Module mmap_index 

Flat binary mmap format for sparse-vector posting lists.

Layout (every struct is #[repr(C)] for a stable layout):

[FileHeader]                    16 bytes
[DimHeader × num_dims]          16 bytes × N
[PostingEntry × total_entries]  16 bytes × M
[Footer]                        8 bytes    (version 2)

Version 3: a dimension is its global token id. DimHeader used to carry a padding word and to be addressed by its position — a dense index local to the file, which only sparse_dims.bin could translate back. The padding word now holds the token_id, the table is sorted by it, and a dimension is looked up by binary search. Same 16 bytes, and three consequences: merging two files is a merge-join on sorted ids with no remapping (so merging two indexes is the same operation), the dims side file is not needed to search, and nothing local to a file leaks into what it means. Versions 1 and 2 keep the dense reading.

Version 2 adds the footer — a CRC-32 of everything before it, then the magic again — and the file is written to a temporary and renamed over the destination. Until 3.0.5 the writer opened the destination itself and wrote in place: an interrupted commit (a crash, a full disk) left a truncated index that opened without complaining and answered wrong. A version 1 file still opens, without those checks.

What is verified at open is the length the headers imply — the cheap check that catches a truncation. The CRC covers the whole file, so verifying it means reading it: MmapPostingData::verify_checksum does that on demand, and LUCIVY_SPARSE_VERIFY_CRC=1 makes every open do it.

Each entry carries a max_next_weight ceiling. Files written here store the inclusive ceiling of the wand module (tail_max: the maximum weight over the entry and everything after it); readers fold max(weight, max_next_weight), so a file whose ceiling excludes the entry itself reads identically.

Structs

MmapPostingData

Mmap’d posting data — read-only view of sparse.mmap.

PostingEntry

On-disk posting entry.

Functions

search_mmap

Top-limit search straight from the mapping, without loading anything into RAM. dim_map is only used for a version 1 or 2 file; a version 3 one carries its own dimensions (see [dims_for]).

search_mmap_allowed

search_mmap restricted to allowed ids (see run_search_allowed).

write_file_atomic

data into path, atomically — the sidecars of a sparse shard (vectors.bin, dims.bin), which fs::write truncated in place just like the postings did. Their bytes are unchanged: no footer, no header, the same bincode a previous version wrote and reads.

write_mmap_file

Write the format from in-RAM posting lists. postings[i] is the list of the dimension whose global token id is dim_tokens[i]; the file is written with its header table sorted by token id, which is what makes two files mergeable without remapping anything.