Module bm25 

BM25 scoring and corpus IDF statistics for hybrid vector+lexical search.

Storage

IdfStats is persisted as a compressed binary blob at <table_root>/metadata/ailake_bm25_stats.bin (zstd-compressed bincode). The path is recorded in Iceberg table properties under ailake.bm25.stats-path so readers know where to find it.

Accuracy

IDF is computed from ALL documents written through TableWriter::write_batch when a bm25_text_column is configured. Concurrent writers may lose DF deltas due to a read-modify-write race on the stats file (same caveat as Iceberg without OCC). Compaction rebuilds stats accurately from all surviving data files.

Structs

BM25Scorer

BM25 scorer backed by global IdfStats.

HybridConfig

Configuration for hybrid vector+BM25 search.

IdfStats

Corpus-level IDF statistics accumulated from all ingested documents.

Enums

HybridFusion

Fusion method for combining vector and BM25 ranked lists.

Constants

BM25_STATS_FILE

Default relative path for the BM25 stats file within the table root.

BM25_STATS_PATH_PROP

Constant used in Iceberg properties to point at the BM25 stats file.

Functions

linear_score

Apply linear fusion over normalized (vec_dist, bm25_score) pairs.

rrf_score

Apply RRF fusion over (vec_rank, bm25_rank) pairs.

tokenize

Tokenize text into lowercase alphanumeric terms, dropping single-char tokens.