Module bm25 

tiny BM25 inverted index for the hybrid search path.

Tokenization is intentionally simple - lowercase, split on Unicode-aware whitespace + punctuation, drop tokens with len < 2. Multilingual corpora (PT-BR, EN, ES) get reasonable behavior without a stemmer or stop list. A future version can pluggable-ize the tokenizer if needed.

The index is small enough that we keep it fully decoded in memory at open time. On-disk layout (section 0x08, payload version 1):

  u32 LE  payload_version = 1
  f32 LE  k1                (BM25 hyperparameter)
  f32 LE  b                 (BM25 hyperparameter)
  f32 LE  avgdl             (average doc length)
  u32 LE  n_docs
  u32 LE  n_terms
  for d in 0..n_docs:
      u32 LE  dl_d          (document length in tokens)
  for t in 0..n_terms:
      u32 LE  len_t
      bytes   token_t       (UTF-8)
      u32 LE  df_t          (document frequency)
      for posting in postings_t:
          u32 LE  doc_id
          u32 LE  tf

Structs

Bm25Index

Constants

BM25_PAYLOAD_VERSION

on-disk payload version for the bm25 section (0x08). v1 stored every posting as raw (u32 doc, u32 tf); v2 delta-gaps the (sorted) doc ids and bitpacks the gaps, the term-frequencies, and the doc lengths with intpack. the decoded index is identical, so scores are unchanged. the reader still accepts v1. the section is optional and excluded from content_hash, so this bump is additive within v1.

DEFAULT_B

DEFAULT_K1

Functions

rrf_union

Reciprocal-rank fusion of two ranked candidate lists. Returns the union sorted by RRF score descending. The runtime then reranks with the exact dot product so the final scores are real cosine.