Struct TextIndex 

pub struct TextIndex { /* private fields */ }

Text index for keyword search with pluggable tokenization and phrase support

Implementations

impl TextIndex

pub fn new() -> Self

Create a new TextIndex with the default SimpleTokenizer

pub fn with_tokenizer(tokenizer: Box<dyn Tokenizer>) -> Self

Create a new TextIndex with a custom tokenizer

pub fn tokenizer_name(&self) -> &'static str

Get the name of the current tokenizer

pub fn index_document(&mut self, id: Id, text: String)

Index a document’s text

pub fn remove_document(&mut self, id: &str)

Remove a document from the index

pub fn bm25_scores(&self, query: &str) -> HashMap<Id, f32>

Compute BM25 scores for query terms

BM25 formula: score(D, Q) = Σ IDF(qi) * (f(qi, D) * (k1 + 1)) / (f(qi, D) + k1 * (1 - b + b * |D| / avgdl))

where:

• IDF(qi) = log((N - df(qi) + 0.5) / (df(qi) + 0.5))
• f(qi, D) = frequency of qi in document D
• |D| = length of document D
• avgdl = average document length
• k1 = 1.2 (term frequency saturation)
• b = 0.75 (length normalization)

pub fn phrase_search(&self, phrase: &str) -> HashMap<Id, f32>

Search for an exact phrase and return matching document IDs with scores

Uses position information to verify that terms appear consecutively. Returns BM25 scores with a phrase boost for exact matches.

pub fn has_text(&self, id: &str) -> bool

pub fn get_text(&self, id: &str) -> Option<&str>

pub fn export_texts(&self) -> &HashMap<Id, String>

Export text data for persistence (Major Issue #6 fix)

Returns the texts HashMap which can be serialized and saved to disk. The inverted index, doc_lengths, and statistics are not exported since they can be rebuilt by re-tokenizing the texts on load.

pub fn import_texts(&mut self, texts: HashMap<Id, String>)

Import text data from disk and rebuild the index (Major Issue #6 fix)

Takes a HashMap of texts loaded from disk and rebuilds the inverted index, doc_lengths, and statistics by re-tokenizing all texts. This allows the text index to survive store reopens and snapshots.

Trait Implementations

impl Default for TextIndex

fn default() -> Self

Returns the “default value” for a type.