Module index 

Task 33 — staged in-repo lexical memory index (spec §9.6 bounds; SQLite declined, see docs/decisions/T33-memory-index-no-sqlite.md).

The index is fully DERIVED state over the append-only JSONL store:

• immutable ts-desc-sorted segment files of doc summaries ({"id","ts","tags","terms"}). Lexical terms are CONTENT-DERIVED — they reveal the vocabulary of record bodies, so index files and directories are private-moded (0600/0700) like the store itself, and secret-sensitivity bodies contribute NO terms at all (metadata-only indexing: id, timestamp, tags);
• an atomically renamed manifest.json recording the consumed store offset, segment metadata (with min/max timestamps for range skipping), and the tombstone set;
• queries stream segment lines through a k-way ts-desc merge with a result-bounded collector — resident memory is proportional to the requested limit plus at most MAX_SEGMENTS buffered docs (staging compacts segments whenever they exceed that bound), never to the total match count, and streaming early-terminates once a page is full;
• crash safety + integrity: torn store tails are healed by the store’s append path and skipped by the lenient reader; orphaned *.tmp files are ignored; the manifest records byte length AND sha256 per segment, and any unparseable manifest, offset mismatch, or missing/truncated/ corrupted segment triggers a VERIFIED full rebuild from the store (the index can always be deleted safely).

No network: this module performs filesystem I/O only. Embeddings are DISABLED — no embedding hook exists here, so no implicit remote call is possible (spec §9.6; local embeddings would be a separate ask-first opt-in).

Structs

IndexHit

A matching document reference (bodies stay in the store; fetch by id).

IndexPage

One bounded result page.

IndexQuery

Index query. Terms use token-exact AND semantics over the lexical term sets (see the T33 decision record for the trade-off vs FTS).

IndexSearchStats

Observability for the proportional-memory bound.

IndexStatus

Status returned by ensure_index_in.

PageCursor

Exclusive pagination cursor: results strictly AFTER this position in (timestamp desc, id desc) order.

Constants

MAX_SEGMENTS

Hard bound on staged segments (and therefore on open readers per query). Staging compacts everything above this into fresh globally ts-ordered segments.

MAX_TERMS_PER_DOC

Lexical terms retained per document (deduped, sorted).

SEGMENT_MAX_DOCS

Docs per immutable segment file.

Functions

ensure_index_in

Stage the index up to the current end of the store: consume unindexed store bytes into new immutable segments and an atomically replaced manifest. Crash-safe at every step; rebuilds from scratch when the manifest is unparseable or the store shrank (rewritten by retention).

index_dir_in

Per-project index directory: <base>/memory/index/<namespace>/.

search_index_in

Search the staged index (auto-staging the tail first). Bounded page, newest-first, resident memory proportional to the limit.