Module text_index 

Inverted text index (TIX1) for fast find_text / recall lexical prefilter.

The hot search paths scan every block (find ~51 ms, recall ~255 ms on ~1M blocks). This index maps lowercased words to the blocks that contain them, so a query only touches the blocks that can possibly match, then the caller re-runs its exact predicate on those candidates (identical results, far fewer text scans).

Binary layout (mmap, zero heap):

  [0..4)   magic "TIX1"
  [4..8)   word_count  u32
  [8..12)  total_postings u32
  [12..16) block_count u32
  [16..20) postings_offset u32 (absolute byte offset; 4-aligned)
  [20..)   dict offsets: word_count × u32 (absolute byte offset of a DictEntry)
  DictEntry (variable):
    u16 word_len + word bytes + u32 posting_start + u32 posting_len
  padding to 4-byte alignment
  postings: total_postings × u32 block ids (each list ascending)

Tokenization is lowercased alphanumeric runs of at least 2 chars. Postings are block ids in ascending order (blocks are visited ascending at build time), so membership checks use binary search.

Structs

TextIndex

Mmap-backed inverted text index.

Functions

build_text_index

Build a TIX1 index from block texts. block_count is the total number of blocks (used for staleness checks at open time).

tokenize

Split text into lowercased alphanumeric words (length >= 2). find_text semantics are substring-based, so query tokens are treated the same way here and candidates are later verified with the exact predicate.