khive-pack-knowledge 0.15.0

Knowledge verb pack — lore corpus (atoms/domains), TF-IDF retrieval, concept registration
Documentation
# khive-pack-knowledge — Design

## Purpose

`khive-pack-knowledge` implements the knowledge corpus verbs for khive. It manages:

- **Corpus tier** — slug-keyed atoms and domain groupings stored in dedicated SQL tables
  (`knowledge_atoms`, `knowledge_domains` — V19 migration)
- **Section tier** — structured subsections per atom (8-value closed enum: overview, core_model,
  boundary_conditions, formalism, operational_guidance, examples, failure_modes, expert_lens)
- **Evaluation tier** — operator-triggered, draft-inclusive retrieval evaluation with summaries
  stored in the pack-owned `knowledge_eval_runs` table
- **KG concept tier** — `learn` / `cite` / `topic` verbs as sugar over the KG entity layer

## ADR Compliance

### Knowledge Pack Verb Surface (ADR-047)

- This pack implements the 21 verb corpus surface: atoms/domains CRUD, TF-IDF search with
  embedding rerank, fold, import, edit, challenge, adjudicate, and concept-tier sugar.
- Domain matching is case-insensitive: domain values are trimmed and lowercased before storage
  and comparison. The same normalized value is used in `properties.domain`, promoted tags,
  and response bodies so all three surfaces agree.

### Section Profiles and Vamana ANN Integration (ADR-048)

- The `SectionType` enum is a closed 8-value set. Headings in atlas markdown files are mapped
  to canonical section types via `from_str_loose`, which accepts common heading aliases; a
  heading that matches no type is not imported and is counted in `sections_unknown_type`.
- The section-read verb surface (Phase 3) is not yet wired. Forward-deployed helpers
  (`section_from_row`, `section_to_json`) are retained so Phase 3 can land without structural
  changes.
- Vamana ANN integration provides a parallel semantic signal to TF-IDF scoring. ANN hits are
  fused via RRF (k=60). The index lifecycle is owned by `knowledge/vamana.rs`: warm-load from
  persistent snapshot, fingerprint validation, rebuild from sqlite-vec corpus on stale/miss.

### Section Review Lifecycle (ADR-047)

- `knowledge.challenge` marks a section as disputed and increments `dispute_count` on the parent
  atom. `knowledge.adjudicate` resolves the dispute: `accept` → `verified`, `reject` → `reviewed`.
  This governance is specified in ADR-047 (Knowledge Pack) §section lifecycle governance.
- The Vamana warm-start protocol (`ensure_ann_background`) uses one `AnnWarmState` lifecycle per
  `{namespace, model}` key. Startup v1/v2 discovery and request background warm
  share the same attempt-owned `begin_warm` / `finish_warm` singleflight; a late
  pre-invalidation completion cannot release or complete a newer attempt.

### Pack Self-Registration (ADR-027)

- `KnowledgePackFactory` is submitted via `inventory::submit!` so the runtime can discover and
  load this pack by name without explicit wiring. `REQUIRES = ["kg"]` declares the dependency.

### Retrieval Quality Measurement (ADR-082)

- `knowledge.eval_retrieval` is an operator-only `Subhandler`, so it does not change the 19-verb
  MCP surface. It searches only atoms at a fixed k of 5 and includes drafts so retrieval quality
  remains independent of finalization coverage.
- Query-set validation is fail-fast. Successful runs persist namespace-scoped aggregate
  precision, recall, and MRR for `knowledge.stats`. Query-set paths must be absolute and are
  canonicalized before reads and persistence so daemon launch directories cannot change a run.

### Edge Ontology (ADR-002)

- `knowledge.cite` creates an `introduced_by` edge from a concept entity to a source entity
  (document, person, or org). The edge direction is concept → source.

### Lexical candidate fetch and stage budget (issue #1930 Amendment 2, issue #2396)

- `fetch_fts_candidates` (`knowledge/search/lexical.rs`) runs each FTS5 term in two phases instead of one
  joined query. Phase A enumerates bare rowids off the `fts_knowledge` index, carrying no namespace
  predicate because
  `namespace` is UNINDEXED on that external-content table and filtering on it there would force a
  content-row fetch per candidate anyway:
  `SELECT rowid FROM fts_knowledge WHERE fts_knowledge MATCH ?1 ORDER BY rowid LIMIT ?2`.
  FTS5 consumes rowid order directly, so enumeration stops at the limit without sorting or
  computing BM25 over the entire match set. TF-IDF remains the application-level ranker.
  When a term exceeds its admitted window (the fetch limit for a single term; up to
  `FTS_TERM_LIMIT`, 500, for a multi-term query), candidates are its lowest eligible rowids,
  not the BM25-best matches admitted by the previous fetch. This trades bounded probe cost
  against recall for high-frequency terms, which carry little IDF weight and run after rarer
  terms under capped-frequency scheduling; eligibility probes can widen to 8000 rowids as below.
  Multi-term queries first probe at most 501 rowids per term and process non-empty terms by
  ascending observed frequency, breaking ties by term spelling. Frequencies up to 500 are exact;
  larger posting lists share the capped frequency. These index-only probes also run inside the
  lexical deadline. Completed terms are merged round-robin, retaining rare matches if a later
  common term times out and preventing query-word order from selecting the surviving prefix.
  Phase B hydrates only the surviving rowids from `knowledge_atoms` in chunks, applying namespace,
  soft-delete, status, and type eligibility at that point — the first query in the whole fetch to
  touch a full atom row (including `content`), and only for rows that already cleared phase A's
  cap. This closes the read-cost hole where every FTS match paid a scattered read against the
  whole (multi-gigabyte-scale) atom table before its own per-term `LIMIT` applied.
  The phase-A and phase-B statements share no snapshot, so a row edited between them is scored
  from its current hydrated text. The hydration statement (`phase_b_hydration_statement`)
  filters on `+a.namespace = ?1` (a unary-plus, not a bare equality) for the same reason as
  the sibling `hydrate_atoms_statement`/
  `hydrate_domains_statement`: this codebase never runs `ANALYZE` on `knowledge_atoms`, so the
  no-statistics planner prefers `idx_knowledge_atoms_ns` over the rowid primary key for a large
  `rowid IN (...)` list; the unary plus defeats that index without changing the predicate.
- When phase B carries a status or type eligibility predicate, phase A overfetches by
  `PHASE_A_OVERFETCH_FACTOR` (4x) so an ineligible-heavy top page cannot starve phase B of rows
  that are eligible further down the rowid sequence. If phase A still returned a full page and
  eligible rows remain short of the per-term cap, the probe widens by the same factor until the
  ceiling, `PHASE_A_WIDEN_CEILING` (8000) — bounding the worst case to a fixed per-term cost instead
  of an unbounded retry loop. If widening reaches the ceiling and the eligible set is _still_
  short, the term falls back once to an eligibility-scoped join (`fts_knowledge` drives primary-key
  lookups in `knowledge_atoms`, with every predicate applied before its own per-term `LIMIT`,
  still in FTS rowid order) so more than `PHASE_A_WIDEN_CEILING` ineligible prefix rows can never hide
  a real candidate — this fallback trades the bounded-cost guarantee for correctness on the rare
  term pathological enough to exhaust the ceiling. Its enumeration can stop once enough eligible
  rows have been found; the independent stage deadline still bounds a long ineligible prefix.
- A genuine lexical miss returns no candidates; unrelated recent rows are never used as a
  fallback. Empty candidates are classified as `filtered` versus `no_match` using namespace-scoped
  evidence. The first membership probe reuses the bounded phase-A rowids. If no local match is
  visible and a term's window was full, a namespace-only FTS existence query checks beyond the
  window, so a foreign prefix cannot change a local filtered match into `no_match`. Its `LIMIT 1`
  bounds returned rows, not scanned work; the same lexical deadline bounds a long prefix or an
  absent local match. Positive eligible matches and fully enumerated misses need no extra query.
- Both `PHASE_A_WIDEN_CEILING` and `LEXICAL_STAGE_BUDGET_MS` (below) have test-only overrides
  (`with_phase_a_widen_ceiling_override`, `with_lexical_stage_budget_override_ms`) implemented as
  `tokio::task_local!` values scoped to the calling task, mirroring
  `khive_storage::scope_request_read_deadline`'s own mechanism — never a process-global `AtomicU64`,
  which a concurrently running test with no override of its own could observe.
- The lexical fetch runs under its own read-deadline budget (`LEXICAL_STAGE_BUDGET_MS`, 2s per
  lexical fetch), scoped via `khive_storage::scope_request_read_deadline`. `search_decomposed`
  runs up to three fetches, so a decomposed request can spend up to three stage budgets on
  lexical work before other stages. Nested deadlines keep whichever is earlier while active
  and pop back to the outer request deadline once the scope exits, so a
  lexical-stage timeout narrows only that stage's own budget — it no longer implies the whole
  request is out of time. `search`, `suggest`, and their decomposed variant gate every downstream
  step (embedding rerank, body-line counts, member-size pricing) on the live ambient
  `khive_storage::request_read_is_cancelled()` check rather than the stage-local timeout flag, so a
  lexical-only degradation with request time left to spare still gets a full rerank pass; the
  `lexical_timeout` degradation flag is still attached to the response whenever the stage's own
  candidate fetch was cut short, independent of whether the rest of the request completed normally.
- One read inside the stage runs under a budget tighter than the stage's: the rarity probe, bounded
  at a quarter of the stage budget. Its expiry is not a cut fetch. The probe produces an ORDERING of
  the terms, every term is queried either way, and the fetch that follows still has the rest of the
  stage budget, so the response carries the rows a build without the optimization would have
  returned. It is reported ADDITIVELY as `degraded.lexical_ordering_probe_timeout`, a boolean set
  whenever any pass fell back to arrival order, BESIDE `degraded.lexical_timeout` rather than
  instead of it.
- `degraded.lexical_timeout` stays COARSE — any lexical timeout sets it — and that is a
  privacy property, not an oversight. Narrowing it to mean "the candidate fetch was cut" would make
  it false for a probe-only expiry and true once a later phase timed out; the later phases are
  reachable only through global-index matches, so the flag would then tell a caller that a row in
  another namespace matched. That is the same fact the operator-only records are withheld from
  `lexical_timeout_details` to protect, and it would be republished as a boolean. The
  ordering-probe flag is safe to add beside it precisely because the probe runs in `term_frequency`,
  a public phase whose entry does not depend on corpus contents. Guarded by
  `mixed_pass_capability_marker_does_not_reveal_foreign_matches`, which requires the public JSON to
  be byte-identical across corpora that differ only in a foreign match. `lexical_timeout_details` stays a SINGLE list covering both, with the population it always
  had: `term_frequency` is one of only two public phases and is the most common timeout by a wide
  margin, so giving the probe a separate list would have emptied the public timing surface for the
  majority of timeouts. A caller separating a fallback from a cut reads `bound` on the records it
  was given, and takes the absence of a stage-bounded record as "not disclosed" rather than as
  "did not happen". Each timeout record carries `bound`, naming which budget governed the read
  (`stage` or `ordering_probe`), and `read_budget_ms`, the budget that governed THAT read — the two
  fields that let a caller separate the cases without losing the timings. `configured_budget_ms`
  and `effective_budget_ms` keep their existing meaning and both describe the stage at its entry,
  which is why a probe-bounded record needs the third number: without it the record reports a read
  cut at 501 ms against a 1999 ms allowance that had not expired (issue #2879).
- `degraded.lexical_timeout` remains a boolean. On instrumented builds, every response carrying ANY
  timeout record also carries `degraded.lexical_timeout_instrumented=true` — including one whose only
  record is an ordering-probe fallback, because the marker identifies build capability and not
  whether rows are missing — regardless of whether any phase is withheld. This marker identifies build capability, not
  hidden execution details; healthy lexical responses omit it entirely, never emitting false.
  When a timed-out read's phase can be disclosed
  without exposing global-index match presence, `degraded.lexical_timeout_details` adds at most
  one record per lexical pass (`full`, `subquery_1`, `subquery_2`), at most three records total.
  Each record contains `pass`, `phase`, `bound`, `stage_elapsed_ms`, `operation_elapsed_ms`,
  `configured_budget_ms`, `effective_budget_ms`, and `read_budget_ms`. Milliseconds are monotonic durations,
  truncated to integers and captured when the failed read returns. The effective budget is the
  remaining deadline allowance at stage entry, including an earlier parent deadline; it is not
  reset after cancellation or sampled after leaving the scope. Stage elapsed includes earlier
  work in the pass, while operation elapsed covers just the failed read attempt, including
  waiting and cleanup. Neither is an attribution of deadline expiry versus cancellation.
  Healthy responses omit the list. Public phases are `reader_open` and `term_frequency`, whose
  entry does not depend on corpus contents. The later `phase_a_rowids`, `phase_b_hydration`,
  `eligibility_fallback`, `namespace_membership`, and `namespace_existence` records are operator-only:
  their reachability can change with foreign-index matches even when local results do not.
  Responses with only operator-only records retain both booleans and omit the details list.
  The public list can omit records and is not a complete execution ledger, even when non-empty.
  All seven phases emit the same structured timing fields through tracing, with no SQL, query
  text, row identifiers, namespace names, row counts, or byte counts. A phase identifies where
  the timeout was observed, not why the allowance was exhausted. Pooled-reader waiting occurs
  inside query calls and cannot be separated from execution without separate backend timing.
  The tracing record also carries `completed_reads`: for each phase that ran a read which did not
  time out, its total milliseconds and read count, in first-run order, so a cut read that ran
  briefly still shows where the rest of the stage went. Read counts in the later phases grow with
  terms, probe widenings and hydration chunks, so like their timings they reflect global-index
  matches; the breakdown is tracing-only and never enters `lexical_timeout_details`.
- `load_domain_member_token_sizes` (member-token pricing for `suggest`'s `results[].size`) returns
  a `(HashMap<String, usize>, bool)` — the `bool` marks whether the whole batch timed out before
  any domain could be measured. A single `query_all` call has no partial-completion state, so one
  flag covers every domain in the batch. The map starts empty, and left joins preserve each live
  domain reached by the sizing query even when it has no live members, measuring that domain at
  zero; missing or deleted members cost zero because compose expands no live content for them.
  A domain absent from the sizing query stays absent from the map and is unmeasured.
  `suggest` never serializes `size: 0` for an unmeasured domain, and it does not serialize a
  `null` size either: the domain is withheld from
  `results` entirely and listed instead under `degraded.member_sizing_timeout.excluded` as
  `{id, name, rank, score}`, so the caller still sees which ranked hit was withheld and why.
  This stable exclusion key covers every unmeasured domain, whether sizing timed out or not.
  `FoldCandidate::size` (`knowledge.fold`) is a non-optional `usize`, so a `null` size would turn
  one unmeasured domain into a hard parse error for a caller that feeds `suggest`'s `results`
  straight into a `fold` call (issue #105's documented passthrough contract). Withholding the
  unpriced domain from `results` keeps that passthrough valid while still refusing to let an
  unpriced domain enter a budgeted fold selection for free.

### Schema Ownership (ADR-015, ADR-028)

- Corpus tables (`knowledge_atoms`, `knowledge_domains`) are added in V19 migration.
- Section table (`knowledge_sections`) is added in a subsequent migration.
- `knowledge_eval_runs` is auxiliary Knowledge-pack state declared through the pack's static and
  runtime schema plans. Centralized startup applies its idempotent DDL to the backend assigned to
  Knowledge; it does not claim a core migration-ledger version.

### ADR-016: Request DSL

- All 21 verbs are accessible through the `request` tool. The public surface is `KnowledgePack`
  only — `handlers` and `knowledge` modules are `pub(crate)`.

## Consistency Notes

- The ANN lifecycle under `knowledge/vamana/` is split by concern: `state.rs` owns `AnnState`,
  its generation counter and the warm-ownership locks; `bridge_impl.rs` implements the bridge;
  `persistence.rs`, `fresh_tail.rs` and `snapshot.rs` hold durable state, the fresh-tail read path
  and snapshot load; `warm_and_rotate.rs` holds warming and segment rotation. The ordering rules
  stay in the functions that take the locks, and every submodule reaches them through the parent.
- The `Section` struct and its associated helper functions (`section_from_row`, `section_to_json`)
  are forward-deployed for Phase 3; they carry `#[allow(dead_code)]` with REASON annotations.

## Module Boundaries

`knowledge/search/ann_search.rs` holds private ANN candidate acquisition, score provenance,
RRF fusion, and post-hydration filters. `search/rerank.rs` holds the query-embedding cache
and reranking; the search parent retains shared record types. `search/lexical.rs` owns lexical candidate selection and budgets;
`search/search_core.rs` combines searches; `search/hydration.rs` joins candidates
with stored records; `search/compose_helpers.rs` loads compose inputs and scores
entity additions; `search/handlers.rs` implements search, suggest, and compose.

| Module                  | Responsibility                                                                           |
| ----------------------- | ---------------------------------------------------------------------------------------- |
| `lib.rs`                | Public exports and the operator-facing `reindex_knowledge` library entry                 |
| `pack.rs`               | Pack registration, `Pack` trait impl, `PackRuntime::dispatch` shim                       |
| `vocab.rs`              | Pack schema statements and the handler descriptor table (21 public verbs + 1 subhandler) |
| `handlers.rs`           | `learn`, `cite`, `topic` verbs (KG concept tier sugar)                                   |
| `knowledge/mod.rs`      | Knowledge handler module boundaries and shared exports                                   |
| `knowledge/eval.rs`     | Offline query-set validation, atom retrieval scoring, and run persistence                |
| `knowledge/schema.rs`   | Param and record types for serde deserialization and SQL row mapping                     |
| `knowledge/vamana.rs`   | Shared Vamana ANN index lifecycle (warm-start, build, search, RRF fusion)                |
| `knowledge/matching.rs` | TF-IDF term matching primitives (tokenize, exact match, count)                           |

## Namespace Isolation

All corpus SQL queries include `AND namespace = ?` predicates scoped to the caller token's
namespace. The `knowledge.import` verb delegates to the import-preserving atom upsert path and
`edit`, which each enforce the caller namespace — no cross-namespace write is possible.
`knowledge.compose`
accepts an explicit `namespace` as ADR-007's exact single-namespace escape. The same derived
token scopes automatic suggestion, corpus/section reads, KG blending, and the cross-pack
brain-profile weight read; Tier-3 pack-local feedback state is keyed by that namespace too.
Nested profile dispatches preserve the authorized token's per-request actor and scope. Direct
pack calls must present a matching authorized token, whose visibility is then narrowed to the
one explicit namespace. An absent parameter preserves the existing caller-token scope.
Other corpus handlers continue to consume the registry's transport-level namespace routing.
The evaluation runner derives the same token namespace for draft-inclusive search and persisted
run summaries; `knowledge.stats` reads only summaries from that namespace.

## Test Coverage

- `tests/integration.rs` — full verb surface, happy path + edge cases
- `tests/fixes.rs` — targeted regression coverage for audit-identified invariants
- `tests/import_integrity.rs` — bounded traversal, stable path identity, validate-first writes,
  whole-file atom mode, and additive import reporting
- `tests/eval_retrieval.rs` — schema routing/idempotence, validation, draft-corpus scoring,
  persistence, and namespace isolation
- `tests/bench.rs` — warm-latency smoke test (ignored by default; see `docs/benchmarks.md`)