Struct InvertedIndex 

pub struct InvertedIndex {
    pub doc_lengths: HashMap<VectorId, usize>,
    pub total_docs: usize,
    pub total_tokens: usize,
    /* private fields */
}

BM25-scored inverted index.

Scoring formula:

IDF(t) = ln((N − df + 0.5) / (df + 0.5) + 1)
score(q, d) = Σ IDF(tᵢ) × tf × (k1 + 1) / (tf + k1 × (1 − b + b × |d| / avgdl))

Fields

doc_lengths: HashMap<VectorId, usize>

total_docs: usize

Total number of indexed documents.

Renamed from doc_count in P23; the serde alias preserves backward-compatible deserialization of existing JSON files that still use "doc_count".

total_tokens: usize

Running sum of all document lengths; enables O(1) avg_doc_length recomputation.

Defaults to 0 for backward compat; load() migrates old files that lack this field.

Implementations

impl InvertedIndex

pub fn new(k1: f32, b: f32) -> Self

pub fn index_document(&mut self, id: &VectorId, text: &str) -> Result<()>

Add or update a document. If the document already exists it is first removed.

pub fn remove_document(&mut self, id: &VectorId) -> Result<()>

Remove a document from the index.

pub fn score( &self, query_terms: &[String], candidate_ids: &[VectorId], ) -> Result<Vec<(VectorId, f32)>>

Score a pre-tokenized list of query terms against a set of candidate documents.

If candidate_ids is empty all documents in the posting lists are scored. Results are sorted descending by BM25 score.

pub fn search(&self, query: &str, k: usize) -> Result<Vec<(VectorId, f32)>>

Tokenize query then score all matching documents. Returns top k.

pub fn score_candidates( &self, query_text: &str, candidate_ids: &[VectorId], ) -> Result<Vec<(VectorId, f32)>>

Tokenize query_text and score against candidate_ids.

If candidate_ids is empty, scores all matching documents. Results sorted descending by BM25 score.

pub fn save(&self, path: &Path) -> Result<()>

Serialize to JSON and write to path.

pub fn load(path: &Path) -> Result<Self>

Load from a JSON file written by save. Reconstructs the tokenizer.

Migration: if total_tokens is 0 (absent in old files) but doc_lengths is non-empty, the running sum is recomputed once via O(n) scan so that subsequent calls to recompute_avg_doc_length are O(1).

pub fn doc_count(&self) -> usize

pub fn avg_doc_length(&self) -> f32

Trait Implementations

impl Clone for InvertedIndex

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InvertedIndex

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InvertedIndex

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for InvertedIndex

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for InvertedIndex

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.