Crate kibble 

kibble as a library: retrieval + ask, plus the full toolkit under the default full feature.

Modules

ask

kibble ask — grounded, cited RAG over the indexed corpus (agentic search loop).

bench

bm25

Hand-rolled BM25 lexical scorer (no external dependency).

build

caps

catalog

classify

kibble auto-topics (Spec B) — assign each catalog document a hierarchical topic from the train answer-cluster space. See docs/CATALOG.md. Opt-in via [classify].enabled.

clean

cluster

Deterministic spherical k-means topic clustering of dataset answers.

codebase

config

corpus

crawl

dataset

dotenv

Minimal .env loader — no external crate.

embed

Embedding backends: an OpenAI-compatible endpoint plus a deterministic stub for tests. Batching is the caller’s job (see vectors::get_or_embed); embed_batch embeds exactly the slice it is given, in order.

eval

extract

fetch

git

index

Retrieval-index corpus loading + chunking. Loads documents from the configured sources (or an explicit path), then chunks them for embedding and BM25 indexing.

ingest

init

kibble init — scaffold a ready-to-run project (config + sample corpus) in the CWD.

llm

Shared OpenAI-compatible chat helper (used by bench and ask).

mcp

mega

minhash

net

normalize

Detect an SFT dataset’s schema and map each record to a {messages} Row.

pack

rebalance

Build-side topic rebalancing: cap over-represented topics by dropping their fringe (farthest-from-centroid) rows, against centroids clustered in-memory from the curated train — which are then persisted as clusters.json (measure == cap).

records

Read any supported data file (jsonl or parquet) as a stream of JSON records. Both formats reduce to Vec<serde_json::Value> so downstream schema detection is format-agnostic.

retrieve

Query the retrieval index: BM25 lexical ranking + semantic cosine ranking, fused by Reciprocal Rank Fusion.

serve

simhash

Random-hyperplane LSH (SimHash) over dense embeddings, mirroring minhash.rs but for cosine space: sign-bit signature -> banded candidates -> exact cosine confirm -> union-find. Deterministic (fixed-seed planes).

soul

text

train

kibble train — run a user-configured trainer command against the tune package produced by kibble tune. Backend-agnostic: the argv is entirely user-specified.

tune

kibble tune — generate the Unsloth sprint curriculum + config.yaml from the built dataset, mapping phases to sources via [tune].phase.

vectors

Persistent, model-scoped vector store + cache. Vectors are keyed by sha256(model ‖ 0x00 ‖ text) so the same answer text embeds once across build and eval. On disk: a packed little-endian f32 blob plus a JSON index ({dim, entries: {hash: row}}).

web

websearch

Shared web-search + page-fetch tools (used by bench research and ask --web).