Module index 

Index construction: turn a corpus of token embeddings into a searchable PLAID index.

build_index ties together the three lower layers:

1. Flatten every document’s token matrix into one big cloud of points.
2. Run k-means to pick coarse centroids (crate::kmeans::fit).
3. Assign every token to a centroid, compute residuals, train cutoffs and weights on the residuals (crate::codec::train_quantizer), and encode each token against the fresh codec.

The resulting Index keeps one EncodedVector per original token plus the per-document doc_id bookkeeping. Inverted-file construction and the query-time search path will be added on top in later TDD cycles.

Structs

DocumentTokens

A single document’s worth of token embeddings, ready to index.

Index

A fully-built PLAID index over a corpus of multi-vector embeddings.

IndexParams

Parameters that control how an Index is built.

InvertedFile

Inverted file: for each centroid, the sorted list of unique document indices that have at least one token clustered in that centroid.

Functions

build_index

Build a Index from a corpus of documents.

build_index_from_pool

Build an Index from a pre-assembled token pool and per-document (doc_id, n_tokens) metadata.