# Retrieve layer — hybrid search (`kibble index` / `kibble search`)
The retrieve layer turns the corpus into a searchable index: `kibble index` chunks and embeds
documents into a BM25 + vector index, and `kibble search` ranks passages against a query by fusing
lexical (BM25) and semantic (cosine) rankings with **Reciprocal Rank Fusion (RRF)**. It reuses the
`understand` layer's embedding backend and vector cache — no new crate dependency, no separate
embedding path — and fails soft to lexical-only search when embeddings aren't available.
## `kibble index [path]`
```bash
kibble index # index [index].sources
kibble index notes/ # index an explicit dir or file instead
```
Builds (or rebuilds) the retrieval index:
1. **Load documents.** With no `path` argument, loads every configured `[index].sources` entry
(default `data/raw`, `data/ingest`, `data/extracted`) that exists on disk — `data/raw` uses the
same corpus loader as `build` (richer doc ids/titles), the rest are walked recursively for
text-like files (`.txt`/`.md`/`.markdown`/`.json`/`.jsonl`), each read whole-file as one
document. Passing `path` (a file or a directory) indexes only that, ignoring `[index].sources`.
2. **Chunk.** Each document is split into passages of up to `[index].chunk_chars` characters
(default 800), preferring to break on paragraph boundaries. `chunk_id` is `"{source}#{n}"`.
3. **Embed (fail-soft).** If `[understand.embed].base_url` is configured, every chunk's text is
embedded through the shared `data/embeddings` vector cache (`[understand.embed]` — same store
`build`/`eval` use, so chunks already embedded elsewhere are reused for free). If no backend is
configured, or the embed call fails, indexing continues and produces a **lexical-only** index —
`kibble index` never errors out over embeddings.
4. **BM25.** A hand-rolled BM25 inverted index is built over the tokenized chunk text, regardless
of whether embeddings succeeded.
5. **Persist** under `[index].dir` (default `data/index/`):
- `chunks.jsonl` — one `Chunk` (`chunk_id, doc_id, source, title, text`) per line.
- `bm25.json` — the serialized BM25 index.
- `meta.json` — `{embed_model, dim, n_chunks, has_vectors}`, used by `search` to detect a
model mismatch or a lexical-only build.
Prints `Indexed N chunk(s) -> data/index (or [index].dir)`.
## `kibble search "<query>" [--k N] [--json]`
```bash
kibble search "how does the retry backoff work"
kibble search "ssrf guard" --k 5 --json
```
Loads the persisted index and ranks chunks against the query:
- **BM25** ranks every chunk by lexical match against the query terms.
- **Semantic** ranks every chunk by cosine similarity of its cached vector to the query's vector —
only when the index has vectors (`meta.has_vectors`), an embed backend is configured, *and* the
configured `[understand.embed].model` matches the model the index was built with. A mismatch
prints a warning and falls back to lexical-only for that query.
- The two rankings (or just BM25, in the fail-soft/lexical-only case) are fused with **Reciprocal
Rank Fusion**: `fused(c) = Σ 1 / (rrf_k + rank(c))` across each ranking chunk `c` appears in,
summed and sorted descending. `rrf_k` is `[index].rrf_k` (default 60). Top `--k` (default 10)
hits are returned.
**Output** — human table by default (rank, RRF score, source, title, 120-char snippet):
```
1. [0.0328] data/raw/hacking/alpha.txt — alpha
first raw document about alpha …
```
`--json` instead prints an array of `{score, source, doc_id, title, text}` (full chunk text, no
snippet truncation) — for piping into other tools or an eventual `kibble ask` / RAG layer.
## Fail-soft / lexical-only behavior
`kibble search` never requires an embedding backend:
- **No embed backend configured** (`[understand.embed].base_url` empty) or the index was **built
lexical-only** (`meta.has_vectors == false`) → BM25 only, no network call.
- **Endpoint unreachable / embed call fails** → a warning is printed to stderr
(`kibble: semantic search skipped (embed failed): ...`) and search continues with BM25 only.
- **Index model mismatch** — the model recorded in `meta.json` differs from the currently
configured `[understand.embed].model` — a warning is printed and search falls back to BM25 only
(comparing vectors across different embedding spaces would be meaningless).
The same fail-soft rule applies to `kibble index`: without a reachable embed backend it just builds
a lexical (BM25-only) index instead of erroring.
**Missing or empty index** — `data/index/chunks.jsonl` doesn't exist, or exists but has no
chunks — `kibble search` errors clearly: `no index found — run \`kibble index\` first` /
`index is empty — run \`kibble index\` first`.
## Config (`[index]`)
```toml
[index]
dir = "data/index" # default
sources = ["data/raw", "data/ingest", "data/extracted"] # default
chunk_chars = 800 # default
rrf_k = 60 # default
```
- `dir` — where the built index (`chunks.jsonl`/`bm25.json`/`meta.json`) is read/written.
- `sources` — paths (relative to the repo root) scanned by `kibble index` when no explicit path is
given; missing entries are skipped.
- `chunk_chars` — max passage length in characters.
- `rrf_k` — the RRF constant; higher values flatten the influence of rank position (favor breadth
across both rankings), lower values weight top ranks more heavily.
Embeddings are configured under `[understand.embed]` — see `docs/UNDERSTAND.md` for the backend,
auth, and vector-store details. There is no separate `[index].embed` block; `index`/`search` reuse
the same backend and the same `data/embeddings` cache as `build`/`eval`.
## Testing
No new crate dependency — BM25 is hand-rolled next to the existing SimHash/MinHash code, and
`index`/`search` reuse the existing embedder trait and vector store. All tests are deterministic
and run offline: BM25 and RRF are pure functions tested directly, and the embed path is exercised
against a `StubEmbedder` (identical text → identical vector) rather than a real endpoint.
## Out of scope (follow-ups)
- **`kibble ask`** — RAG answer synthesis (retrieve + generate) on top of `kibble search`.
- **Web-augmentation** — blending live web search results into retrieval (`kibble bench`'s
`method = "research"` already does something similar for benchmarking; this would bring it to
everyday search/ask).
- **ANN index** — the semantic ranking is currently exact (brute-force cosine over all chunk
vectors), fine at corpus sizes tested so far; an approximate nearest-neighbor index would be a
follow-up for much larger corpora.