# Changelog
All notable changes to `kibble`, a **mass-knowledge & data platform** — consume anything, extract
and structure it, and turn it into clean datasets and knowledge bases at scale (discover →
acquire → extract → understand → curate → retrieve → tune). Entries are grouped by milestone;
the project is pre-1.0 and unversioned.
## Auto-taxonomy (`[classify]`, Spec B)
- **Opt-in emergent document topics.** With `[classify].enabled = true`, `build` tags every catalog
document with a hierarchical `auto_topic` (e.g. `Systems › Linux › Memory`) + `topic_confidence`,
and writes `data/catalog/topics.json`. It reuses the train answer-cluster space (rebalance's
in-memory centroids when available — matching `clusters.json` — else clustered fresh), LLM-names each
cluster hierarchically, and assigns each document the majority topic of its rows (ties → lowest cluster
index). Default off → build stays offline and `documents.jsonl` byte-identical. Fail-soft (missing/
erroring embed backend, no `[ask]` LLM, or empty centroids → auto-topics skipped, rest of catalog
intact); reporting-only (not read by `index`/`ask`); LLM naming is non-deterministic. No new
dependency. (Spec A shipped the complete inventory; Spec B makes it self-organizing.)
## Catalog completeness (Spec A)
- **`build` catalogs every source, not just `data/raw`:** `data/catalog/documents.jsonl` and
`summary.json` now contain one entry per document for **all** source types — Files/Web/Blog (the
`Document` each already builds) and Dataset/Codebase (one synthesized entry per `doc_id`, classified
on the rows' non-`system` text). Codebase entries default to `role: "code"` unless `source_defaults`
or a per-doc override sets one. Reporting-only — the catalog is not read by `index`/`ask` (retrieval
runs off `[index].sources`), so this changes the inventory, not what gets retrieved. No new
dependency. (Spec B will add emergent auto-topics.)
## `kibble train` — run the trainer against the tune package (Tune B)
- **`kibble train`** runs the user-configured `[train].command` (an argv list, no shell) against the
`kibble tune` package: it validates the package (`config.yaml` + at least one `sprint/<phase>/
train.jsonl`), substitutes placeholders (`{config}`/`{out_dir}`/`{sprint_dir}`/`{model}` → absolute
paths; unknown token = error), then spawns the command with `cwd` defaulting to `[tune].out`,
streaming its output and propagating its exit code. `--dry-run` prints the resolved command and
runs nothing; `-- <extra>` appends substituted args. Backend-agnostic — worked examples for
unsloth/Kaggle, MLX, and generic commands in `docs/TRAIN.md`, which also carries the **Tune C
runbook** (upload → train → download adapter → `kibble bench` the tuned model). No new dependency.
## Dogfood cleanups (#43)
- **`build` counts all sources' documents:** `stats.total_documents` and `documents_by_source` now
count distinct input documents across `data/raw` **and** every `[[source]]` (Files/Dataset/Codebase/
Web/Blog) — previously they counted only the `data/raw` pile, so a `[[source]]`-only build reported
`0 docs`.
- **`cluster` guards degenerate tiny topics:** `[cluster].min_cluster_size` (default `2`) merges any
cluster *smaller than* the threshold into its nearest neighbor after k-means (deterministic; `0`/`1`
= off), so a too-high `k` no longer leaves singleton topics (raise the threshold to also fold larger
runts — e.g. `3` merges size-1/2 clusters). The post-merge topic count can be lower than `k`. No new
dependency.
## `bench --stream` — live research output
- **`kibble bench --stream`** streams the model's output to **stderr** during `method="research"`
benchmark runs, for live visibility. Research-only (non-research methods are single-shot and
unaffected); **the scored answer, score, `--strict` gate, and the printed results table are
byte-identical** with or without the flag — only live stderr output is added (the streamed
turn's latency is measured locally, same as buffered; `report.json`'s wall/latency fields vary
run-to-run either
way). Opt-in, default off. Reuses `llm::chat_turn_stream`; no new dependency (#21).
## `ask --json --stream` — NDJSON event streaming
- **`kibble ask --json --stream`** now streams the answer as newline-delimited JSON events —
`{"type":"search",…}` for each agentic retrieval, `{"type":"token","text":…}` as the model
generates, then a terminal `{"type":"answer",…}` (same fields as the buffered `--json` object) or
`{"type":"notfound",…}`. Lets programmatic consumers stream an agentic RAG answer; unlike text
streaming it does not require a TTY. `--json` alone is unchanged (single buffered object). No new
dependency (#22).
## `kibble tune` — generate the training curriculum from the build
- **`kibble tune`** turns the built dataset into the Unsloth trainer's phased **sprint curriculum**
(`sprint/<phase>/train.jsonl`) + `config.yaml`, mapping each `[[tune.phase]]` to a set of sources.
`build` now emits a per-row provenance sidecar (`<split>.sources.jsonl`; `train.jsonl` stays
messages-only), which `tune` reads to split rows by source. `config.yaml` is written as JSON
(valid YAML for the trainer's `yaml.safe_load`). Closes the `build → tune → pack → train` loop;
training orchestration + adapter-back/eval are deferred follow-ups. No new dependency. `docs/TUNE.md`.
## LLM-named cluster labels — `[cluster].llm_labels`
- **`[cluster].llm_labels = true`** (opt-in) names each topic via the `[ask]` LLM from its
distinctive terms + a sample (`0-old-css` → `Old vs modern CSS`), replacing the term-based
labels in `clusters.json`, the `kibble cluster` report, and eval's topic breakdown. Applies
everywhere `clusters.json` is written (command, build auto-cluster, rebalance). One LLM call
per run; **fail-soft** — any problem keeps the distinctive-term labels. Default off (zero LLM
calls). Centroids/sizes/assignment stay deterministic; only label text varies when on.
Reuses the `[ask]` endpoint; no new dependency (#26).
## `ask` always shows its grounding
- **`kibble ask` now surfaces the retrieved passages** when a (weak local) model answers from the
corpus but emits no `[n]` citation markers — instead of `0 source(s)`, it prints a
`References (retrieved — the model didn't mark citations)` block (top 3 by score) and a
state-aware footer (`0 cited · N retrieved · M search`). General-knowledge answers
(`--chance` / not-found "try anyway") carry a sharpened `⚠ Not in your corpus …, verify
independently.` banner, and the JSON output gains a bounded `retrieved` array. Found by
dogfooding (#35), where a correct grounded answer showed zero sources with a 4B local model.
Display-layer only — retrieval and the agentic loop are unchanged.
## Rebalancing converges in one build (inline / Approach B)
- **`[cluster].rebalance` now clusters the curated train in-memory**, caps over-represented
topics against those centroids, and writes them as `data/clusters.json` — so the eval
`max_topic_share` gate and the rebalance cap read the **same** centroids and it converges
in a single build (fixes #34, where the old "cap against the prior clusters.json, then
re-cluster" path oscillated/diverged). The first-build no-op is gone. With rebalance on,
`clusters.json` reflects **train-only** topics; off, it clusters all splits as before.
One embed + one clustering per build; no new dependency. Replaces the Approach-A
mechanism from #31.
## Topic-balance gate — `eval` reads `clusters.json`
- **`kibble eval` adds a `max_topic_share` metric** (stage C of clustering, #25): it assigns each
**train** answer to its nearest topic centroid from `data/clusters.json` and gates on the largest
topic's share of the training set. Auto-runs when `clusters.json` + an embed backend are present;
**fail-soft** otherwise (missing file, **model mismatch**, dimension mismatch, or embed error → the
metric is skipped and `eval` still completes). Counts toward the quality score / `overall` and is
gated by `--strict` with no extra flag. Reuses the `data/embeddings` cache (embeds raw train
answers). Config: `[eval.thresholds].max_topic_share` (default `0.60`). No new dependency.
Build-side topic-aware rebalancing remains a follow-up.
## Topic clustering — `kibble cluster`
- **`kibble cluster [--k N] [--json]`** clusters the built dataset's rows on their answer embeddings
(deterministic spherical **k-means** + k-means++ seeding, no new dep), labels each topic by its
**distinctive terms**, prints a breakdown, and writes `data/clusters.json` (centroids + labels).
**Auto-runs during `kibble build`** (`[cluster]` config, `k=12` default; fail-soft — skipped without
an embed backend). Centroids are the seam for a future `eval` topic-balance gate (#25); LLM-named
labels are a follow-up (#26). Reuses the `data/embeddings` cache. `docs/CLUSTER.md`.
- Labels filter a small static **stopword list** so grammar particles (`the`, `and`, `to`, …)
never dominate a topic label.
## `ask` token streaming — `kibble ask` streams live
- `kibble ask` now **streams the answer token-by-token** to the terminal as the model generates it
(`[ask].stream`, default on; `--stream`/`--no-stream`; active only on a TTY and not `--json`).
During tool rounds it prints stderr activity notes (`⋯ searching corpus …`). New `llm::chat_turn_stream`
(SSE via `reqwest::Response::chunk()` — no new dep) accumulates content + tool-call deltas and returns
the same shape as `chat_turn`, so citations/loop are unchanged. Follow-ups: bench streaming (#21),
`--json` streaming (#22).
## Web-augmented `ask` — `kibble ask --web`
- **`kibble ask "<q>" [--web] [--no-web]`** gives the model `web_search` + `fetch_page` tools
(shared `websearch` module, SearXNG or DuckDuckGo) alongside `search_corpus`, so it answers
from the corpus first and the live web when needed. Citations tag **[local]** vs **[web]**
(`--json` sources gain `"kind"`). `[web]` config (`default`, `base_url`, `max_results`,
`allow_hosts`, fetch caps); web off = corpus-only (unchanged). `fetch_page` stays SSRF-guarded
+ capped. No new crate dep; offline tests via localhost mocks. `bench` reuses the same `websearch` module.
## Ask layer — agentic grounded RAG (`kibble ask`)
- **`kibble ask "<q>" [--k N] [--json] [--chance] [--show-context]`** answers questions grounded in
the indexed corpus: it seeds a `search`, lets the served LLM call a `search_corpus` tool up to
`[ask].max_rounds` to dig further, then returns a **cited** answer (`[n]` → source) or, when the
corpus doesn't cover it, an interactive not-found flow (provide a source & retry / add detail /
`--chance` general-knowledge fallback, flagged). Reuses the retrieval index + a shared `llm` chat
helper (extracted from `bench`). Needs a served LLM (`[ask].base_url`); retrieval stays fail-soft.
No new crate dep; deterministic offline tests via a scripted-chat seam. `docs/ASK.md`.
## Retrieve layer — hybrid search (`kibble index` / `kibble search`)
- **`kibble index [path]`** builds a retrieval index over the corpus (`data/raw`/`data/ingest`/
`data/extracted`, or an explicit path): chunks each document, embeds chunks via the shared
`data/embeddings` cache (`[understand.embed]`), and builds a hand-rolled **BM25** inverted index →
`data/index/{chunks.jsonl,bm25.json,meta.json}`.
- **`kibble search "<q>" [--k N] [--json]`** ranks passages by **Reciprocal Rank Fusion** of semantic
cosine + BM25, returning score + provenance. **Fail-soft:** with no/unreachable embed backend it
runs lexical-only (BM25). No new crate dep; deterministic offline tests. `docs/RETRIEVE.md`.
## Understand layer — embeddings, semantic dedup & leakage
- New `kibble` embedding backend (`[understand.embed]`, OpenAI-compatible `/embeddings`) with a
content-hash **vector store/cache** (`data/embeddings/`, reused across `build`→`eval`) and a
hand-rolled **SimHash LSH** (random hyperplanes → cosine confirm → union-find). No new crate dep.
- **Semantic dedup** in `build` (`[curate].semantic_dedup`, opt-in): collapses near-paraphrase
answers, keeping the longest; reported as `dropped_semantic_duplicates`.
- **Semantic metrics** in `eval` (`[eval].semantic`, opt-in): `semantic_near_duplicate_rate` +
`semantic_leakage_rate`, gated (defaults 0.02 / 0.0 at cosine 0.90) into the score and `--strict`.
- Backend: `llama-server --embedding` (nomic-embed-text, 768-dim) on localhost. Deterministic,
offline tests via a stub embedder. `docs/UNDERSTAND.md`.
## `.env` auto-loading
- kibble now loads the nearest **`.env`** (searching the working directory upward) into the process
environment at startup, so secrets (`OCR_API_KEY`, `KIBBLE_API_TOKEN`, `OPENAI_API_KEY`, …) no
longer need manual `source`-ing. Real environment variables always win (a shell `export`
overrides `.env`). Hand-rolled parser (`KEY=VALUE`, `#` comments, optional `export ` prefix,
quote-stripping) — **no new crate dep**; prints a one-line notice when it loads anything. Ships
`.env.example`; `.env` is gitignored.
## Soul / serving-artifact emitter — `kibble soul build`
- **`kibble soul build [--soul] [--out]`** compiles a versioned `soul.toml` (persona + system prompt(s)
+ sampling + tools + optional base template) into serving artifacts: `generation_config.json`,
`system_prompt.txt`, `tools.json`, `chat_template.jinja` (marker substitution — `{{SOUL_DEFAULT_SYSTEM}}`
→ JSON-escaped prompt), and `manifest.json` (sha256 — the registry seed). Pure/deterministic, no new
deps; ships `soul.example.toml` + `docs/SOUL.md`. Automates/versions the hand-assembled serving bundle.
## Web crawler — `kibble crawl`
- **`kibble crawl <url> [--depth][--max-pages][--out][--all-hosts]`** — a Firecrawl-style discovery
layer: BFS-crawls a site (same-host default, depth/page limits, `robots.txt` honored, politeness
delay, per-page byte cap, SSRF-guarded, real `User-Agent` sent) and writes each page as **markdown**
into `data/ingest/crawl-<host>/` for `build`. `[crawl]` config; `docs/CRAWL.md`.
- Static HTML only (no JS rendering — follow-up). Reuses `web`/`serve`-guard/`scraper`; no new deps.
Verified against live sites.
## MCP server — `kibble mcp`
- **`kibble mcp`** runs a stdio JSON-RPC 2.0 server (hand-rolled, no MCP SDK dep) exposing the full
measure/emit loop as tools an agent/harness (or Claude Code/Desktop) can call: `build`, `eval`,
`bench`, `soul_build`, `caps_list`, `caps_install`. `initialize`/`tools/list`/`tools/call`/`ping` +
notifications + correct error codes (`-32601`/`-32602`/`-32700`); tool failures → `isError`.
- The harness **integration surface** (the harness itself is a separate project). stdio = local-trust,
no listener; URL sources reuse the fetch SSRF guards; results are JSON. `docs/MCP.md` has client
registration. No new crate deps.
## Web-research / tool-execution bench — `method = "research"`
- Adds an **agentic** benchmark method: the model calls `web_search` / `fetch_page`, the harness
executes them (SearXNG JSON when `[bench.search].base_url` set, else a DuckDuckGo scrape with
`uddg` decode; page fetch via `web::fetch_url_capped` + `extract_main_text`), feeds results back
over the OpenAI tools protocol, and the **final** answer is scored (contains/judge) plus
**turns / tool-calls / wall-time** (the measurable "beat-Perplexity"). `max_turns` cap; per-case
faults score 0 (never abort).
- **Security:** `fetch_page` is SSRF-guarded (`serve::url_is_allowed` + `[bench.search].allow_hosts`
bypass — model-chosen URLs can't reach loopback/private) and byte-capped. No new deps.
## Model benchmark harness — `kibble bench`
- **`kibble bench [--strict]`** benchmarks a *served* model (any OpenAI-compatible endpoint) against
an extensible suite: `[[bench.benchmark]]` + a `cases.jsonl`, scored by `contains` / `mcq` /
`judge`, with per-request **latency + tokens/sec**. Per-benchmark thresholds → overall PASS +
0–100 score; `--strict` exits nonzero for CI. Writes `bench/report.{json,md}`; ships a starter
suite + `docs/BENCH.md`; keys env-only; panic-safe on malformed endpoint responses.
- This is the self-improvement loop's steering signal ("did the model get better / fast enough?").
KIBBLE is the tool, not the harness/model. A live web-research / tool-execution
("beat-Perplexity") benchmark is the flagship follow-up.
## Active curation — `build` produces gate-passing datasets
- **Dedup + leakage-safe split:** `build` now collapses rows with the same normalized answer and
splits by **answer content** (`split_bucket(answer_key)`) instead of `doc_id`, so identical
answers can never land in two splits — exact train↔valid/test leakage is impossible by
construction. `stats.json` gains `dropped_duplicates`. On by default; `[curate]` (`dedup`,
`leakage_safe_split`) to tune. Consolidated the per-source split/stat logic into one
`curate_split` stage.
- On the real 59,632-row corpus: **duplicate 8.5% → 0%, leakage 14.1% → 0%** (5,694 dupes dropped).
- **Curation filters:** `build` also drops malformed (non-SFT-valid), degenerate (empty/echo), and
too-short rows (`[curate]` `drop_malformed`/`drop_degenerate`/`min_answer_chars`, default
on/on/20), reported as `dropped_filtered`. The degeneracy check is clean-aware (matches the
post-`clean_text` content `eval` reads), so a filtered build clears eval's
`malformed_rate`/`degenerate_rate`/`short_rate` by construction (verified: real build → all 0).
- Fixed `eval`'s balance metric to read `stats.json`'s object-shaped `sources` (was silently null).
- **Eval near-dup metrics** (opt-in `[eval].near_dup`): MinHash `near_duplicate_rate` + `near_leakage_rate`, gated, so the quality gate certifies fuzzy dedup/leakage (build fixes, eval certifies).
- **Opt-in fuzzy near-dup** (`[curate].near_dedup`, default off): a deterministic MinHash + LSH +
union-find pass collapses near-paraphrase/boilerplate answers (char k-grams, threshold 0.85),
reported as `dropped_near_duplicates`. Conservative on real prose (~0.6% dropped, raises
`distinct2`); no new deps.
- `eval` `distinct2` perf: hashes bigrams to u64 instead of storing String pairs — full 53k-row
eval went from timing out (>180s) to ~45s.
- Next (deferred): source rebalancing; DPO/pretraining formats; the MCP/harness track.
## Repositioning — mass knowledge & data
- Dropped the original personal "voice/style" framing. KIBBLE is a general knowledge & data
acquisition and curation platform; voice tuning was only the first use-case. Docs reframed.
- `eval` **repurposed** from a voice-fidelity scorer into a **dataset-quality gate**:
`kibble eval [--strict]` reads the built `train/valid/test.jsonl` + `stats.json`, validates SFT
readiness, and scores duplication, train↔valid/test leakage, length, source balance, degeneracy,
and diversity → `report.{json,md}` with a 0–100 score; `--strict` exits nonzero on FAIL so CI can
block a bad dataset. The voice scorer (`VoiceProfile`/LLM judge) is removed. Dataset-shaping
prompt defaults de-voiced. (Real-corpus run caught 14% leakage + 8.5% dupes.)
## Capability registry — Layer 7.1 (agentic harness, skills-first)
Skill installation applied to the agent itself: consume a source → gain its skills.
- **`kibble caps install|scan|list|remove`** — detect `SKILL.md` skills in a source (local path or
URL; URL reuses `fetch::resolve`), install them under `<root>/skills/<name>/`, and maintain
`<root>/registry.json`. `[caps].root` (default `~/.kibble/caps`), `--force`, `--project`.
- Detection prunes nested skills (a skill dir is atomic); frontmatter `name`/`description` with
dir-name fallback; names sanitized.
- Install is **copy + register only** (skill scripts never executed): dests confined under the
skills root, all symlinks skipped (no escape, no recursion), cumulative byte cap, sha256 per
skill, atomic registry write, collision skip-unless-`--force`.
- Foundation for the rest of the harness layer (MCP exposure of the registry, capability
assembly, autonomous build/tune loop + emry).
## Extractor — Layer 3
Turn any acquired artifact into clean text, staged under `data/extracted/` for `build`
(`fetch → extract → build`). Each backend tries a configured OpenAI-compatible endpoint, then a
local CLI, then errors. Keys from env only; CLIs via arg-vectors with **canonicalized input
paths** (no shell, no argv-flag injection); inputs size-capped.
- **3.3 — video + office:** `ffmpeg` extracts a video's audio → transcription; `pandoc` converts
office docs → plain text. `extract` now covers all artifact kinds (no unsupported gap).
- **3.2 — PDF:** text-layer extraction via `pdf-extract`; scanned/sparse PDFs render pages with
`pdftoppm` → OCR each → concatenate. Errors (instead of emitting empty) if every page fails.
- **3.1 — framework + OCR + transcription:** `kibble extract` + artifact detection; image OCR
(vision endpoint, `tesseract` fallback); audio transcription (`/audio/transcriptions`,
`whisper` fallback); html/text passthrough.
## Ingester — Layer 2 (acquire)
- **mega.nz:** download + decrypt public file **and** folder links — hand-rolled MEGA protocol
(AES-128 ECB/CBC/CTR, key derivation, node-tree walk) with no heavy crate; offline known-answer
tested.
- **3c — file hosts:** Dropbox (`dl=1`), Google Drive (`uc?export`), MediaFire (scrape), gofile
(guest-token API).
- **3b — `kibble serve`:** axum HTTP API — `POST /ingest` → job id, `GET /jobs/{id}`, `GET /health`.
Bearer-token auth (fail-closed), an **SSRF guard** (blocks loopback/private/link-local incl.
IPv4-mapped IPv6 and userinfo tricks; `allow_hosts` bypass), in-memory job registry.
- **3a — `kibble fetch`:** pluggable handler layer — git clone, HuggingFace dataset, open HTTP
directory crawl, direct HTTP file, archive extraction (zip / tar.gz, zip-slip & tar-symlink
safe). Streamed `max_bytes` cap; traversal-safe filenames; proxy-aware.
## Eval & Kaggle wiring
- **`kibble eval`:** blended voice-fidelity score — deterministic heuristic profile + a
configurable LLM judge (Anthropic or OpenAI-compatible) anchored on real held-out posts;
writes `report.json` + `report.md`. Generate-or-consume samples; keys from env only;
judge failures non-fatal.
- **`kibble pack`:** package the build output into a Kaggle dataset bundle (`[pack]` config:
splits, includes, metadata) + zip; `kaggle/RUNBOOK.md` documents `build → pack → upload → train`.
- **Polish:** configurable eval `reference_max`; auth header only sent when a key is set
(local endpoints need none); `refresh = true` updates a cached git clone via `git pull`.
## Build pipeline & config
- **`[paths]` config:** `data_root` / `dataset_dir` configurable (defaults preserve the layout).
- **Phase 2.6:** `extract_main_text` refinements (`<pre>` preserved, nested-block de-dupe);
host-anchored github detection; `Files` (`.txt`/`.md` dir) and `Blog` (`.html`) source types;
dataset `preserve_code` (bypass the prose cleaner for code-bearing datasets).
- **Phase 2.5 (a–c):** tokio runtime + proxy-aware `net` client + `kibble.toml` source spine;
external chat-dataset merge; local codebase scan; git-URL clone; web/URL fetch + main-text
extraction.
- **Phase 2 — `kibble build`:** raw docs + configured sources → catalog (role/topic) → chunk →
deterministic 78/12/10 split (`SHA256(doc_id)`) → `{messages}` JSONL + `stats.json`; code
bypasses the prose cleaner, prose is cleaned at write.
## Foundation
- **Phase 1 — `kibble clean` + `kibble ingest`:** the text-cleaning rules (strip
mentions/URLs/hashtags/tickers/subreddits/invites/zero-width/empty-brackets, emails preserved)
and local ingest (Twitter GDPR archive, textfiles tree).
---
Cross-cutting throughout: Rust orchestrator routing to external backends; everything
offline-tested (localhost mocks + stub binaries, no real network/tools in tests); responsible
acquisition (SSRF/size guards). Design specs and implementation plans live in
`docs/superpowers/` in the source project.