Module vectors 

Vector recall for the Stop step (internal evaluation “vectors”, not published).

Word matching (BM25) shortlists the lessons the consequence model reads at a turn end. It misses lessons that share no rare word with the work: paraphrased or abstract ones, German work against English lessons, renamed identifiers. Vectors add those, and only those:

• Where: only on the Stop path, after the free gate passed and right before the consequence model call (seconds), where loading the embedder is small next to it. Prompt, early delivery and the free gate stay BM25 only.
• How: no rank fusion. The shortlist is BM25’s top 20 as before, plus up to k vector hits not already in it, under the same scope rule and done set (extend). The model decides relevance; vectors only add recall. R3b’s widening stays BM25-based.
• Index: an embedded SeekStorm vector index per store (<store>/vectors/), one vector per lesson (question + situations + consequence), updated after borg learn / borg assimilate --apply, after a mesh sync brings new lessons, and by borg vectors --rebuild; a missing index is built by the first update. A lesson without a vector is simply BM25-only.
• Model: Model2Vec potion-multilingual-128M from a local directory (BORG_VECTOR_MODEL), fetched once by borg vectors --fetch; never downloaded inside a hook. The embedding matrix is mapped, not read: a query touches only the rows of its own tokens.
• Fail open: no model, no index, a locked index or any error = BM25 only. On by default: borg doctor warns about a missing model or index (doctor.rs); BORG_VECTORS=off disables it; a lean build (--no-default-features) never has it.

Re-exports

pub use model::MODEL_FILES;

pub use model::MODEL_REPO;

pub use model::MODEL_REVISION;

pub use model::enabled;

pub use model::index_dir;

pub use model::model_dir;

pub use model::model_present;

Modules

doctor

What borg doctor and borg install --apply say about vector recall (vectors.rs): the model, and each store’s index. Vectors are on by default, so a missing model or index is a WARN with its fix; off by choice (BORG_VECTORS=off) or a lean build is fine. Read-only: nothing here fetches or builds.

model

Where vector recall’s model and indexes live and whether they are there, without loading either: what the hooks check before a query, and what borg doctor reads (doctor.rs). No dependency on lessons or the engine.

Constants

DEFAULT_K

Vector hits added to the shortlist at most (BORG_VECTOR_K). Baseline: the internal evaluation “vectors” setting; with the whole-store pool (whole.rs) the shortlist only orders what fills the budget when the pool overflows.

SEGMENT_BITS

SeekStorm’s segment number bits of the index (2^11 segments). Method: SeekStorm’s own default for an index of this size; borg-search’s evaluation builds its indexes with the same layout.

SHORTLIST

BM25 lessons the Stop’s consequence model sees before vector additions. Baseline: top 30 found as much as top 20 plus vectors (internal evaluation “vectors”); with the whole-store pool it is the overflow order’s head (whole.rs select).

Functions

extend

The Stop shortlist: bm25 (BM25’s top 20, as select_scoped returns it) and then, in vector order, up to k lessons of ranked (lesson ids, best first) that bm25 lacks. Each added lesson must be in pool (the lessons not done this session), in scope for scope_text (rank_scoped’s rule), not superseded and not dormant (a dormant lesson needs a strong word match, which a vector hit is not).

k

How many vector hits the Stop step adds at most (BORG_VECTOR_K, default DEFAULT_K).

lesson_text

The text a lesson is embedded from: question, situations, consequence.

main

borg vectors: status of the model and the store’s index; --rebuild (re)builds the index; --fetch downloads the model (pinned revision, sha256 checked); --query (reached only as borg debug vectors --query) prints the Stop step’s shortlist for a context on stdin (offline checks, latency). The exit code.

query_tail

What the consequence model reads of the work (predict.rs prompt): the last TAIL characters.

recall

The ids of the lessons nearest to text, best first, at most n; empty whenever vectors are not available (fail open: no model, no index, the index being written, any error or panic).

refresh

Bring the index of lessons up to date after lessons were written (borg learn, borg assimilate --apply, a mesh sync), building it if it is missing (the first lesson after --fetch needs no --rebuild). Quiet and fail-open: without the model or with vectors off it does nothing; an error is one line on stderr. Returns the number of lessons embedded.

shortlist

The Stop step’s shortlist for the lessons file lessons: BM25’s top n over context (scoped by scope_text) extended by the vector index’s hits (extend). Without vectors (off, no model, no index, any error) it is exactly BM25’s top n. Also returns how many lessons the vectors added.