Struct Index 

pub struct Index {
    pub params: IndexParams,
    pub codec: ResidualCodec,
    pub doc_ids: Vec<u64>,
    pub doc_centroid_ids: Vec<u32>,
    pub doc_residual_bytes: Vec<u8>,
    pub doc_offsets: Vec<usize>,
    pub ivf: InvertedFile,
}

A fully-built PLAID index over a corpus of multi-vector embeddings.

Stored in a flat, StridedTensor-style layout — every document’s encoded tokens live in one of two big contiguous buffers, doc_centroid_ids (one u32 per token) and doc_residual_bytes (packed_bytes_per_token u8s per token). Per-document slicing happens through the precomputed doc_offsets cumulative lengths.

This layout matches fast-plaid’s StridedTensor and buys three things over the older Vec<Vec<EncodedVector>> we used to keep:

1. Search doesn’t have to walk per-document Vecs on every query to gather codes and residuals — it slices into the flat buffer.
2. Peak RAM drops: 6.8M tokens at nbits=2 was ~500 MiB of heap for the old nested-Vec headers; the flat layout is ~240 MiB.
3. GPU decode can Tensor::from_slice the whole slice for a batch of candidates in one kernel launch instead of one-per-token.

Callers that want the per-token view use Index::doc_centroid_ids / Index::doc_residual_bytes — slices into the flat buffers with no allocation.

Fields

params: IndexParams

codec: ResidualCodec

doc_ids: Vec<u64>

doc_centroid_ids: Vec<u32>

Flat [total_tokens] vector of per-token centroid indices.

doc_residual_bytes: Vec<u8>

Flat [total_tokens * packed_bytes_per_token] residual bytes, row-major — each packed_bytes_per_token-long slice is one token’s packed residual.

doc_offsets: Vec<usize>

Cumulative per-document token counts; length num_docs + 1, doc_offsets[i + 1] - doc_offsets[i] = n_tokens for doc i.

ivf: InvertedFile

Centroid → tokens inverted file used for candidate generation.

Implementations

impl Index

pub fn from_encoded_docs( params: IndexParams, codec: ResidualCodec, doc_ids: Vec<u64>, doc_tokens: Vec<Vec<EncodedVector>>, ivf: InvertedFile, ) -> Self

Construct an Index from per-document EncodedVectors.

Convenience for tests, crate::update::apply_update, and crate::persistence’s legacy-format loader: they still think in terms of Vec<Vec<EncodedVector>>, and this flattens that into the canonical Index layout in one pass.

ivf should already reflect the doc_tokens contents — this helper does not recompute the inverted file, only the flat per-token buffers.

pub fn num_documents(&self) -> usize

Number of documents currently stored in the index.

pub fn num_tokens(&self) -> usize

Total number of encoded tokens across all documents.

pub fn position_of(&self, doc_id: u64) -> Option<usize>

Find the position of a document inside Index::doc_ids.

pub fn doc_token_count(&self, idx: usize) -> usize

Number of encoded tokens for the idx-th document.

pub fn doc_centroid_ids(&self, idx: usize) -> &[u32]

Slice of per-token centroid indices for the idx-th document.

pub fn doc_residual_bytes(&self, idx: usize) -> &[u8]

Slice of packed residual bytes for the idx-th document, row-major [n_tokens, packed_bytes_per_token].

pub fn doc_tokens_vec(&self, idx: usize) -> Vec<EncodedVector>

Reconstruct the idx-th document’s tokens as an owned Vec<EncodedVector>.

Allocation-heavy — prefer Index::doc_centroid_ids / Index::doc_residual_bytes on the hot path. Provided as a compatibility shim for crate::update::apply_update and legacy callers that still work in EncodedVector terms.

Trait Implementations

impl Clone for Index

fn clone(&self) -> Index

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Index

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.