Struct Bm25Index 

pub struct Bm25Index { /* private fields */ }

Hand-rolled Okapi BM25 index over a set of enriched documents.

Built once via Bm25Index::build; queried repeatedly via Bm25Index::score. Document order matches the chunk-index convention used elsewhere in the ripvec port.

Implementations

impl Bm25Index

pub fn build(chunks: &[CodeChunk]) -> Self

Build an index over enriched chunks. Tokenization uses crate::encoder::ripvec::tokens::tokenize.

Three-pass build:

1. par_iter (tokenize + intern + TF): each chunk is enriched, tokenized, and its tokens interned into a shared ThreadedRodeo. The per-doc TF map keys on the Spur ID instead of String, eliminating the duplicated-string storage that dominated memory + hashing in the previous version.
2. serial DF merge: walk per-doc TF maps and increment a global Spur-keyed counter. With Spur keys (4-byte NonZeroU32), FxHash lookups are a single multiply.
3. serial IDF compute: produce the final df_idf map.

On a 92K-file linux corpus (~250K chunks): bm25_build drops from 35s serial → ~14s parallel without interning → ~7s with interning.

pub fn len(&self) -> usize

Number of indexed documents.

pub fn is_empty(&self) -> bool

Whether the index has zero documents.

pub fn score(&self, query: &str) -> Vec<f32>

Compute BM25 scores for query against every document. Returns a Vec<f32> of length self.len() (one score per doc). Zero scores indicate no query terms matched.

Postings-list scoring: walks postings[term] for each query term (typically <1% of corpus). Per-term work is dispatched via rayon: each thread accumulates a local scores vector, all vectors fold-reduce at the end. Parallelism is bounded by the number of distinct query terms; for the common 1-5-term query rayon uses 1-5 workers, which is appropriate — the algorithmic win from inversion dwarfs any further parallel scaling.