# khive-pack-knowledge — Design
## Purpose
`khive-pack-knowledge` implements the knowledge corpus verbs for khive. It manages:
- **Corpus tier** — slug-keyed atoms and domain groupings stored in dedicated SQL tables
(`knowledge_atoms`, `knowledge_domains` — V19 migration)
- **Section tier** — structured subsections per atom (10-value closed enum: overview, core_model,
boundary_conditions, formalism, operational_guidance, examples, failure_modes, expert_lens,
references, other)
- **Evaluation tier** — operator-triggered, draft-inclusive retrieval evaluation with summaries
stored in the pack-owned `knowledge_eval_runs` table
- **KG concept tier** — `learn` / `cite` / `topic` verbs as sugar over the KG entity layer
## ADR Compliance
### Knowledge Pack Verb Surface (ADR-047)
- This pack implements the 19 verb corpus surface: atoms/domains CRUD, TF-IDF search with
embedding rerank, fold, import, edit, challenge, adjudicate, and concept-tier sugar.
- Domain matching is case-insensitive: domain values are trimmed and lowercased before storage
and comparison. The same normalized value is used in `properties.domain`, promoted tags,
and response bodies so all three surfaces agree.
### Section Profiles and Vamana ANN Integration (ADR-048)
- The `SectionType` enum is a closed 10-value set. Headings in atlas markdown files are mapped
to canonical section types via `from_str_loose`, which accepts common heading aliases.
- The section-read verb surface (Phase 3) is not yet wired. Forward-deployed helpers
(`section_from_row`, `section_to_json`) are retained so Phase 3 can land without structural
changes.
- Vamana ANN integration provides a parallel semantic signal to TF-IDF scoring. ANN hits are
fused via RRF (k=60). The index lifecycle is owned by `knowledge/vamana.rs`: warm-load from
persistent snapshot, fingerprint validation, rebuild from sqlite-vec corpus on stale/miss.
### Section Review Lifecycle (ADR-047)
- `knowledge.challenge` marks a section as disputed and increments `dispute_count` on the parent
atom. `knowledge.adjudicate` resolves the dispute: `accept` → `verified`, `reject` → `reviewed`.
This governance is specified in ADR-047 (Knowledge Pack) §section lifecycle governance.
- The Vamana warm-start protocol (`ensure_ann_background`) uses one `AnnWarmState` lifecycle per
`{namespace, model}` key. Startup v1/v2 discovery and request background warm
share the same attempt-owned `begin_warm` / `finish_warm` singleflight; a late
pre-invalidation completion cannot release or complete a newer attempt.
### Pack Self-Registration (ADR-027)
- `KnowledgePackFactory` is submitted via `inventory::submit!` so the runtime can discover and
load this pack by name without explicit wiring. `REQUIRES = ["kg"]` declares the dependency.
### Retrieval Quality Measurement (ADR-082)
- `knowledge.eval_retrieval` is an operator-only `Subhandler`, so it does not change the 19-verb
MCP surface. It searches only atoms at a fixed k of 5 and includes drafts so retrieval quality
remains independent of finalization coverage.
- Query-set validation is fail-fast. Successful runs persist namespace-scoped aggregate
precision, recall, and MRR for `knowledge.stats`. Query-set paths must be absolute and are
canonicalized before reads and persistence so daemon launch directories cannot change a run.
### Edge Ontology (ADR-002)
- `knowledge.cite` creates an `introduced_by` edge from a concept entity to a source entity
(document, person, or org). The edge direction is concept → source.
### Lexical candidate fetch and stage budget (issue #1930 Amendment 2, issue #2396)
- `fetch_fts_candidates` (`knowledge/search.rs`) runs each FTS5 term in two phases instead of one
joined query. Phase A enumerates bare rowids off the `fts_knowledge` index, carrying no namespace
predicate because
`namespace` is UNINDEXED on that external-content table and filtering on it there would force a
content-row fetch per candidate anyway:
`SELECT rowid FROM fts_knowledge WHERE fts_knowledge MATCH ?1 ORDER BY rowid LIMIT ?2`.
FTS5 consumes rowid order directly, so enumeration stops at the limit without sorting or
computing BM25 over the entire match set. TF-IDF remains the application-level ranker.
When a term exceeds its admitted window (the fetch limit for a single term; up to
`FTS_TERM_LIMIT`, 500, for a multi-term query), candidates are its lowest eligible rowids,
not the BM25-best matches admitted by the previous fetch. This trades bounded probe cost
against recall for high-frequency terms, which carry little IDF weight and run after rarer
terms under capped-frequency scheduling; eligibility probes can widen to 8000 rowids as below.
Multi-term queries first probe at most 501 rowids per term and process non-empty terms by
ascending observed frequency, breaking ties by term spelling. Frequencies up to 500 are exact;
larger posting lists share the capped frequency. These index-only probes also run inside the
lexical deadline. Completed terms are merged round-robin, retaining rare matches if a later
common term times out and preventing query-word order from selecting the surviving prefix.
Phase B hydrates only the surviving rowids from `knowledge_atoms` in chunks, applying namespace,
soft-delete, status, and type eligibility at that point — the first query in the whole fetch to
touch a full atom row (including `content`), and only for rows that already cleared phase A's
cap. This closes the read-cost hole where every FTS match paid a scattered read against the
whole (multi-gigabyte-scale) atom table before its own per-term `LIMIT` applied.
The phase-A and phase-B statements share no snapshot, so a row edited between them is scored
from its current hydrated text. The hydration statement (`phase_b_hydration_statement`)
filters on `+a.namespace = ?1` (a unary-plus, not a bare equality) for the same reason as
the sibling `hydrate_atoms_statement`/
`hydrate_domains_statement`: this codebase never runs `ANALYZE` on `knowledge_atoms`, so the
no-statistics planner prefers `idx_knowledge_atoms_ns` over the rowid primary key for a large
`rowid IN (...)` list; the unary plus defeats that index without changing the predicate.
- When phase B carries a status or type eligibility predicate, phase A overfetches by
`PHASE_A_OVERFETCH_FACTOR` (4x) so an ineligible-heavy top page cannot starve phase B of rows
that are eligible further down the rowid sequence. If phase A still returned a full page and
eligible rows remain short of the per-term cap, the probe widens by the same factor until the
ceiling, `PHASE_A_WIDEN_CEILING` (8000) — bounding the worst case to a fixed per-term cost instead
of an unbounded retry loop. If widening reaches the ceiling and the eligible set is _still_
short, the term falls back once to an eligibility-scoped join (`fts_knowledge` drives primary-key
lookups in `knowledge_atoms`, with every predicate applied before its own per-term `LIMIT`,
still in FTS rowid order) so more than `PHASE_A_WIDEN_CEILING` ineligible prefix rows can never hide
a real candidate — this fallback trades the bounded-cost guarantee for correctness on the rare
term pathological enough to exhaust the ceiling. Its enumeration can stop once enough eligible
rows have been found; the independent stage deadline still bounds a long ineligible prefix.
- A genuine lexical miss returns no candidates; unrelated recent rows are never used as a
fallback. Empty candidates are classified as `filtered` versus `no_match` using namespace-scoped
evidence. The first membership probe reuses the bounded phase-A rowids. If no local match is
visible and a term's window was full, a namespace-only FTS existence query checks beyond the
window, so a foreign prefix cannot change a local filtered match into `no_match`. Its `LIMIT 1`
bounds returned rows, not scanned work; the same lexical deadline bounds a long prefix or an
absent local match. Positive eligible matches and fully enumerated misses need no extra query.
- Both `PHASE_A_WIDEN_CEILING` and `LEXICAL_STAGE_BUDGET_MS` (below) have test-only overrides
(`with_phase_a_widen_ceiling_override`, `with_lexical_stage_budget_override_ms`) implemented as
`tokio::task_local!` values scoped to the calling task, mirroring
`khive_storage::scope_request_read_deadline`'s own mechanism — never a process-global `AtomicU64`,
which a concurrently running test with no override of its own could observe.
- The lexical fetch runs under its own read-deadline budget (`LEXICAL_STAGE_BUDGET_MS`, 2s per
lexical fetch), scoped via `khive_storage::scope_request_read_deadline`. `search_decomposed`
runs up to three fetches, so a decomposed request can spend up to three stage budgets on
lexical work before other stages. Nested deadlines keep whichever is earlier while active
and pop back to the outer request deadline once the scope exits, so a
lexical-stage timeout narrows only that stage's own budget — it no longer implies the whole
request is out of time. `search`, `suggest`, and their decomposed variant gate every downstream
step (embedding rerank, body-line counts, member-size pricing) on the live ambient
`khive_storage::request_read_is_cancelled()` check rather than the stage-local timeout flag, so a
lexical-only degradation with request time left to spare still gets a full rerank pass; the
`lexical_timeout` degradation flag is still attached to the response whenever the stage's own
candidate fetch was cut short, independent of whether the rest of the request completed normally.
- One read inside the stage runs under a budget tighter than the stage's: the rarity probe, bounded
at a quarter of the stage budget. Its expiry is not a cut fetch. The probe produces an ORDERING of
the terms, every term is queried either way, and the fetch that follows still has the rest of the
stage budget, so the response carries the rows a build without the optimization would have
returned. It is reported ADDITIVELY as `degraded.lexical_ordering_probe_timeout`, a boolean set
whenever any pass fell back to arrival order, BESIDE `degraded.lexical_timeout` rather than
instead of it.
- `degraded.lexical_timeout` stays COARSE — any lexical timeout sets it — and that is a
privacy property, not an oversight. Narrowing it to mean "the candidate fetch was cut" would make
it false for a probe-only expiry and true once a later phase timed out; the later phases are
reachable only through global-index matches, so the flag would then tell a caller that a row in
another namespace matched. That is the same fact the operator-only records are withheld from
`lexical_timeout_details` to protect, and it would be republished as a boolean. The
ordering-probe flag is safe to add beside it precisely because the probe runs in `term_frequency`,
a public phase whose entry does not depend on corpus contents. Guarded by
`mixed_pass_capability_marker_does_not_reveal_foreign_matches`, which requires the public JSON to
be byte-identical across corpora that differ only in a foreign match. `lexical_timeout_details` stays a SINGLE list covering both, with the population it always
had: `term_frequency` is one of only two public phases and is the most common timeout by a wide
margin, so giving the probe a separate list would have emptied the public timing surface for the
majority of timeouts. A caller separating a fallback from a cut reads `bound` on the records it
was given, and takes the absence of a stage-bounded record as "not disclosed" rather than as
"did not happen". Each timeout record carries `bound`, naming which budget governed the read
(`stage` or `ordering_probe`), and `read_budget_ms`, the budget that governed THAT read — the two
fields that let a caller separate the cases without losing the timings. `configured_budget_ms`
and `effective_budget_ms` keep their existing meaning and both describe the stage at its entry,
which is why a probe-bounded record needs the third number: without it the record reports a read
cut at 501 ms against a 1999 ms allowance that had not expired (issue #2879).
- `degraded.lexical_timeout` remains a boolean. On instrumented builds, every response carrying ANY
timeout record also carries `degraded.lexical_timeout_instrumented=true` — including one whose only
record is an ordering-probe fallback, because the marker identifies build capability and not
whether rows are missing — regardless of whether any phase is withheld. This marker identifies build capability, not
hidden execution details; healthy lexical responses omit it entirely, never emitting false.
When a timed-out read's phase can be disclosed
without exposing global-index match presence, `degraded.lexical_timeout_details` adds at most
one record per lexical pass (`full`, `subquery_1`, `subquery_2`), at most three records total.
Each record contains `pass`, `phase`, `bound`, `stage_elapsed_ms`, `operation_elapsed_ms`,
`configured_budget_ms`, `effective_budget_ms`, and `read_budget_ms`. Milliseconds are monotonic durations,
truncated to integers and captured when the failed read returns. The effective budget is the
remaining deadline allowance at stage entry, including an earlier parent deadline; it is not
reset after cancellation or sampled after leaving the scope. Stage elapsed includes earlier
work in the pass, while operation elapsed covers just the failed read attempt, including
waiting and cleanup. Neither is an attribution of deadline expiry versus cancellation.
Healthy responses omit the list. Public phases are `reader_open` and `term_frequency`, whose
entry does not depend on corpus contents. The later `phase_a_rowids`, `phase_b_hydration`,
`eligibility_fallback`, `namespace_membership`, and `namespace_existence` records are operator-only:
their reachability can change with foreign-index matches even when local results do not.
Responses with only operator-only records retain both booleans and omit the details list.
The public list can omit records and is not a complete execution ledger, even when non-empty.
All seven phases emit the same structured timing fields through tracing, with no SQL, query
text, row identifiers, namespace names, row counts, or byte counts. A phase identifies where
the timeout was observed, not why the allowance was exhausted. Pooled-reader waiting occurs
inside query calls and cannot be separated from execution without separate backend timing.
The tracing record also carries `completed_reads`: for each phase that ran a read which did not
time out, its total milliseconds and read count, in first-run order, so a cut read that ran
briefly still shows where the rest of the stage went. Read counts in the later phases grow with
terms, probe widenings and hydration chunks, so like their timings they reflect global-index
matches; the breakdown is tracing-only and never enters `lexical_timeout_details`.
- `load_domain_member_token_sizes` (member-token pricing for `suggest`'s `results[].size`) returns
a `(HashMap<String, usize>, bool)` — the `bool` marks whether the whole batch timed out before
any domain could be measured. A single `query_all` call has no partial-completion state, so one
flag covers every domain in the batch. The map starts empty, and left joins preserve each live
domain reached by the sizing query even when it has no live members, measuring that domain at
zero; missing or deleted members cost zero because compose expands no live content for them.
A domain absent from the sizing query stays absent from the map and is unmeasured.
`suggest` never serializes `size: 0` for an unmeasured domain, and it does not serialize a
`null` size either: the domain is withheld from
`results` entirely and listed instead under `degraded.member_sizing_timeout.excluded` as
`{id, name, rank, score}`, so the caller still sees which ranked hit was withheld and why.
This stable exclusion key covers every unmeasured domain, whether sizing timed out or not.
`FoldCandidate::size` (`knowledge.fold`) is a non-optional `usize`, so a `null` size would turn
one unmeasured domain into a hard parse error for a caller that feeds `suggest`'s `results`
straight into a `fold` call (issue #105's documented passthrough contract). Withholding the
unpriced domain from `results` keeps that passthrough valid while still refusing to let an
unpriced domain enter a budgeted fold selection for free.
### Schema Ownership (ADR-015, ADR-028)
- Corpus tables (`knowledge_atoms`, `knowledge_domains`) are added in V19 migration.
- Section table (`knowledge_sections`) is added in a subsequent migration.
- `knowledge_eval_runs` is auxiliary Knowledge-pack state declared through the pack's static and
runtime schema plans. Centralized startup applies its idempotent DDL to the backend assigned to
Knowledge; it does not claim a core migration-ledger version.
### ADR-016: Request DSL
- All 19 verbs are accessible through the `request` tool. The public surface is `KnowledgePack`
only — `handlers` and `knowledge` modules are `pub(crate)`.
## Consistency Notes
- `knowledge/vamana.rs` also exceeds 700 lines by design: the ANN lifecycle (SharedAnn type,
snapshot persistence, build, search) is tightly coupled through the shared `AnnState`
generation and warm-ownership locks and cannot be split without obscuring their ordering.
- The `Section` struct and its associated helper functions (`section_from_row`, `section_to_json`)
are forward-deployed for Phase 3; they carry `#[allow(dead_code)]` with REASON annotations.
## Module Boundaries
| `lib.rs` | Public exports and the operator-facing `reindex_knowledge` library entry |
| `pack.rs` | Pack registration, `Pack` trait impl, `PackRuntime::dispatch` shim |
| `vocab.rs` | Pack schema statements and the handler descriptor table (19 public verbs + 1 subhandler) |
| `handlers.rs` | `learn`, `cite`, `topic` verbs (KG concept tier sugar) |
| `knowledge/mod.rs` | Knowledge handler module boundaries and shared exports |
| `knowledge/eval.rs` | Offline query-set validation, atom retrieval scoring, and run persistence |
| `knowledge/schema.rs` | Param and record types for serde deserialization and SQL row mapping |
| `knowledge/vamana.rs` | Shared Vamana ANN index lifecycle (warm-start, build, search, RRF fusion) |
| `knowledge/matching.rs` | TF-IDF term matching primitives (tokenize, exact match, count) |
## Namespace Isolation
All corpus SQL queries include `AND namespace = ?` predicates scoped to the caller token's
namespace. The `knowledge.import` verb delegates to the import-preserving atom upsert path and
`edit`, which each enforce the caller namespace — no cross-namespace write is possible.
`knowledge.compose`
accepts an explicit `namespace` as ADR-007's exact single-namespace escape. The same derived
token scopes automatic suggestion, corpus/section reads, KG blending, and the cross-pack
brain-profile weight read; Tier-3 pack-local feedback state is keyed by that namespace too.
Nested profile dispatches preserve the authorized token's per-request actor and scope. Direct
pack calls must present a matching authorized token, whose visibility is then narrowed to the
one explicit namespace. An absent parameter preserves the existing caller-token scope.
Other corpus handlers continue to consume the registry's transport-level namespace routing.
The evaluation runner derives the same token namespace for draft-inclusive search and persisted
run summaries; `knowledge.stats` reads only summaries from that namespace.
## Test Coverage
- `tests/integration.rs` — full verb surface, happy path + edge cases
- `tests/fixes.rs` — targeted regression coverage for audit-identified invariants
- `tests/import_integrity.rs` — bounded traversal, stable path identity, validate-first writes,
whole-file atom mode, and additive import reporting
- `tests/eval_retrieval.rs` — schema routing/idempotence, validation, draft-corpus scoring,
persistence, and namespace isolation
- `tests/bench.rs` — warm-latency smoke test (ignored by default; see `docs/benchmarks.md`)