Expand description
Save and load a built Index to/from disk.
The on-disk format is a single little-endian binary file with a small header followed by a sequence of plain f32/u32/u64 blobs. The layout is deliberately boring — there’s no compression, no extra framing, and no schema evolution today; we can layer any of that on top later without changing the semantic API.
Layout, every field little-endian:
magic : 8 bytes, b"PLAIDIDX"
version : u32 (currently 1)
dim : u32
nbits : u32
k_centroids : u32
max_kmeans_iters: u32
n_documents : u64
centroids : dim * k_centroids f32
bucket_cutoffs : 2^nbits - 1 f32
bucket_weights : 2^nbits f32
doc_ids : n_documents u64
token_counts : n_documents u32 (tokens per document)
encoded_tokens : for each token, u32 centroid_id then `dim` u8 codesThe inverted file is not persisted — it’s derivable from the encoded tokens in O(n_tokens) time on load and storing it would just duplicate state that’s already on disk elsewhere.