# Dataset quality (`kibble eval`)
`kibble eval` grades a built dataset's training-readiness and quality, writing
`eval/report/{report.json,report.md}`. It reads `train/valid/test.jsonl` + `stats.json` from
`[eval].dataset_dir` (default `[paths].dataset_dir`).
## Metrics (SFT)
- **readiness / malformed_rate** — every row is valid SFT (`messages`, proper roles, non-empty turns)
- **duplicate_rate** — exact + normalized duplicate answers within train
- **leakage_rate** — valid/test answers that also appear in train
- **short_rate / length** — assistant-content length distribution + too-short fraction
- **degenerate_rate** — empty / echo / too-short rows
- **distinct2** — answer bigram diversity (templated-corpus floor)
- **max_source_share** — largest source's fraction of rows (from `stats.json`)
- **max_topic_share** — largest topic's fraction of **train** rows, via nearest-centroid assignment against `data/clusters.json` (skipped if clusters unavailable)
## Topic balance (`max_topic_share`)
When `data/clusters.json` exists (from `kibble cluster`) and an embed backend is configured via `[understand.embed].base_url`, `eval` measures dataset **topic balance**: it embeds each **raw train answer**, assigns it to its nearest cluster centroid, and gates on the largest topic's share of the training set. The `max_topic_share` metric counts toward the quality score / overall PASS and is enforced by `--strict` exactly like every other metric — no separate flag.
**Note:** `kibble build` can *reduce* topic imbalance via inline rebalancing (`[cluster].rebalance`, default off): it clusters the curated train in-memory, caps over-represented topics, and writes `clusters.json` so `eval` reads the same centroids and converges in one build. The build cap is distinct from the eval gate; set the cap below the gate so rebalancing makes the gate pass. See `docs/CLUSTER.md#cap-vs-gate` and `docs/CLUSTER.md#rebalancing` for details.
**Auto-when-available & fail-soft:**
Computed only when all of the following hold:
- `data/clusters.json` is present (written by `kibble cluster` or a previous `kibble build` with `[cluster].enabled`)
- `[understand.embed].base_url` is set (embed backend configured; an unreachable backend is handled by the embed-error fail-soft path below, not this guard)
- Train split is non-empty
- The cluster model matches `[understand.embed].model` (**model-match guard**: computed embeddings must be comparable to stored centroids)
- Centroid and embedding dimensions match (**dimension guard**: no mismatch from model changes)
If any guard fails or embedding errors occur (network, parse, etc.), the metric is **skipped** (returns `None`) and `eval` completes unaffected — the dataset still gets a quality score and report. Dimension mismatches and model mismatches are printed to stderr so you can rebuild.
**Implementation:** reuses the shared `data/embeddings` cache (embeds raw train answers via the same backend as other eval metrics), then runs nearest-centroid assignment (argmax cosine similarity, ties to lowest index) and counts topic shares.
**Config:**
```toml
[eval.thresholds]
max_topic_share = 0.60 # default: largest topic may be at most 60% of train
```
## Verdict & gate
Each metric is checked against `[eval.thresholds]`. **Overall PASS iff every gated metric is ok**;
`quality_score = 100 × ok / total`. Run with `--strict` to exit nonzero on FAIL so CI/automation can
block a bad dataset:
```bash
kibble eval # report only
kibble eval --strict # exit 1 if the dataset fails the gate
```
```toml
[eval]
dataset_dir = "data/datasets/unsloth"
method = "sft"
[eval.thresholds]
max_duplicate_rate = 0.02
max_leakage_rate = 0.0
max_short_rate = 0.10
max_source_share = 0.60
max_topic_share = 0.60 # gated only when data/clusters.json + an embed backend are present
min_distinct2 = 0.30
```
Active curation that *fixes* a failing dataset (dedup/filter/rebalance in `build`), and DPO/
pretraining formats + fuzzy near-dup, are tracked as follow-ups.