Struct SearchIndex 

pub struct SearchIndex {
    pub chunks: Vec<CodeChunk>,
    pub hidden_dim: usize,
    /* private fields */
}

Pre-computed embedding matrix for fast re-ranking.

Stores all chunk embeddings as a contiguous [num_chunks, hidden_dim] ndarray matrix. Re-ranking is a single BLAS matrix-vector multiply.

When constructed with a cascade_dim, also stores a truncated and re-normalized [num_chunks, cascade_dim] matrix for two-phase MRL cascade search: fast pre-filter at reduced dimension, then full-dim re-rank of the top candidates.

Fields

chunks: Vec<CodeChunk>

All chunks with metadata.

hidden_dim: usize

Hidden dimension size.

Implementations

impl SearchIndex

pub fn new( chunks: Vec<CodeChunk>, raw_embeddings: &[Vec<f32>], cascade_dim: Option<usize>, ) -> Self

Build an index from embed_all output.

Flattens the per-chunk embedding vectors into a contiguous Array2 for BLAS-accelerated matrix-vector products at query time.

When cascade_dim is Some(d), also builds a truncated and L2-re-normalized [N, d] matrix for two-phase MRL cascade search. The truncated dimension is clamped to hidden_dim.

Panics

Panics if the flattened embedding data cannot form a valid [num_chunks, hidden_dim] matrix (should never happen when embeddings come from embed_all).

pub fn rank(&self, query_embedding: &[f32], threshold: f32) -> Vec<(usize, f32)>

Rank all chunks against a query embedding.

Returns (chunk_index, similarity_score) pairs sorted by descending score, filtered by threshold.

pub fn rank_turboquant( &self, query_embedding: &[f32], top_k: usize, threshold: f32, ) -> Vec<(usize, f32)>

TurboQuant-accelerated ranking: compressed approximate scan → exact re-rank.

1. Estimate inner products for ALL vectors via TurboQuant (~5× faster than BLAS).
2. Take top pre_filter_k approximate candidates.
3. Re-rank with exact FP32 dot products on the full embedding matrix.

Falls back to Self::rank when no compressed index is available.

pub fn rank_cascade( &self, query_embedding: &[f32], top_k: usize, threshold: f32, ) -> Vec<(usize, f32)>

Two-phase MRL cascade ranking: fast pre-filter then full re-rank.

1. Layer-norms the query over its full dimension, truncates to truncated_dim, L2-normalizes, and computes dot products against the truncated matrix to find the top pre_filter_k candidates.
2. Re-ranks those candidates using full-dimension dot products.

Falls back to Self::rank when no truncated matrix is available.

pub fn embedding(&self, idx: usize) -> Option<Vec<f32>>

Return a clone of the embedding vector for chunk idx.

Returns None if idx is out of bounds.

pub fn find_duplicates( &self, threshold: f32, max_pairs: usize, ) -> Vec<(usize, usize, f32)>

Find duplicate or near-duplicate chunks by pairwise cosine similarity.

Computes embeddings @ embeddings.T (a single BLAS GEMM) to get all pairwise similarities, then extracts pairs above threshold from the upper triangle (avoiding self-matches and symmetric duplicates).

Returns (chunk_a, chunk_b, similarity) sorted by descending similarity. Each pair appears only once (a < b).

pub fn len(&self) -> usize

Number of chunks in the index.

pub fn is_empty(&self) -> bool

Whether the index is empty.

pub fn truncated_dim(&self) -> Option<usize>

The truncated dimension used for cascade pre-filtering, if enabled.