Expand description
Save and load a built Index to/from disk.
The on-disk format is a single little-endian binary file with a small header followed by a sequence of plain f32/u32/u64 blobs. The layout is deliberately boring — no compression, no extra framing — so that every section can be read or written as one bulk copy.
Layout (version 3), every field little-endian:
magic : 8 bytes, b"PLAIDIDX"
version : u32 (currently 3)
dim : u32
nbits : u32
k_centroids : u32
max_kmeans_iters: u32
n_documents : u64
centroids : dim * k_centroids f32
bucket_cutoffs : 2^nbits - 1 f32
bucket_weights : 2^nbits f32
doc_ids : n_documents u64
token_counts : n_documents u32 (tokens per document)
centroid_ids : total_tokens u32
residuals : total_tokens * (dim * nbits / 8) u8The inverted file is not persisted — it’s derivable from the encoded tokens in O(n_tokens) time on load and storing it would just duplicate state that’s already on disk elsewhere.