Module text 

2h: full-text search as key discipline (FACT-03).

A newcomer’s map: an inverted index answers “which documents contain this word?” – it inverts documents into (word -> documents). Here the btree IS that index: the sorted term keys are the dictionary (prefix search = a range scan), and postings live in two shapes:

• the HEAD (segment 0): one row per (term, doc), written BLIND at index time – searchable the instant it lands, no build step;
• FOLDED segments (1..): one value per term with packed (docid-delta, tf) varints – compact, immutable, produced by folding the head (2h fold, bulk-dock shape).

BM25 in one breath: score = idf(term) * tf / (tf + k1*(1-b+b*|d|/avg)). idf rewards rare words, the denominator saturates repeated words and normalises by document length. Everything it needs lives in keyed rows: per-(field,doc) token counts (0x0D) and per-(field,seg) totals (0x0E).

Structs

PackedTextCandidate

A fully sorted and logically verified text segment that has not touched the shared tree. It is Send because it owns only run-file descriptors and scalar manifests; D26 publication still requires a caller-owned Graph.

SegMeta

Per-(field, segment) manifest. The logical hashes are computed from the source postings before the candidate is written and recomputed by the independently reopened verifier. Counts catch truncation; the two differently seeded hashes catch changed/reordered logical records.

TextBuildAccumulator

Disk-first accumulator for an initial index build. A fixed 4 MiB table combines repeated (raw term number, exact word) records before a fixed 4 MiB external sort. Finishing resolves the vanishingly rare number collision exactly and writes dictionary rows in key order.

TextBuildCache

A fixed-size, direct-mapped accelerator used only while initially backfilling an index. It never grows with the corpus; long words bypass it so its retained memory is below one MiB.

Constants

BM25_B

BM25_K1

Functions

read_varint

tokenize

Lowercase alphanumeric tokens; everything else separates. Deliberately simple (no stemming, no stopwords) – linguistic layers stack on later without changing the keyspace.

write_varint

Varint (LEB128) helpers for packed postings.