Struct TextIndex 

pub struct TextIndex { /* private fields */ }

Mmap-backed inverted text index.

Implementations

impl TextIndex

pub fn open(path: &Path) -> Option<Self>

Open an existing text_index.bin (sparse validation; corrupt files return None so callers fall back to a full scan).

pub fn block_count(&self) -> usize

Number of blocks the index was built over.

pub fn word_count_public(&self) -> usize

Number of dictionary words (distinct tokens) in the index.

pub fn candidates_substring(&self, query_lower: &str) -> Option<Vec<u32>>

Exact superset of blocks whose text contains query as a substring.

Returns None when the query cannot be routed through the index (too short / punctuation-only) — caller must fall back to a full scan. Returns Some(ids) with ids sorted ascending; the caller verifies the exact substring predicate on the candidates.

pub fn candidates_lexical(&self, tokens: &[String]) -> Option<Vec<u32>>

Blocks matching any token under token_similarity semantics: t == q for short tokens (3-4 bytes), or sharing the first 5 chars for tokens of length >= 5. Used as a recall prefilter; None → full scan fallback.

pub fn candidates_lexical_min_matches( &self, tokens: &[String], min_matches: u32, ) -> Option<Vec<u32>>

Like TextIndex::candidates_lexical, but keeps only blocks that at least min_matches distinct query terms matched.

The unbounded union is unusable on a large corpus. A five-character token expands to every dictionary word sharing those five characters and contributes its whole posting list, and on the SciFact index a query about diabetes returned 83,703 blocks – 90% of the query’s time went into scoring blocks that matched one incidental word.

Counting distinct matched terms is the cheapest useful bound available here, because the postings are already being walked: it adds an increment per posting instead of a second pass, and a block that matched four of the query’s terms is a far better candidate than one that matched the single most common word in the corpus.

min_matches == 1 returns the same set as candidates_lexical, and a query with no tokens still returns None, so this is a strict generalisation rather than a second behaviour. Callers are still expected to fall back to the vector side when the result is empty – the lexical index is a prefilter, not a gate.