# Topic clustering — `kibble cluster`
`kibble cluster` groups the built dataset's answer rows into topics based on their embeddings, discovering structure and coverage gaps. It performs deterministic **spherical k-means** clustering, labels each topic by its most distinctive terms (no LLM by default; opt-in LLM naming via `[cluster].llm_labels`), and writes a `clusters.json` artifact that feeds into a future dataset-balance gate.
## What it clusters
Cluster operates on the **built dataset's answer rows**: after `kibble build` writes `train/valid/test.jsonl`, cluster reads all answer texts, looks up their embeddings from the shared `data/embeddings` cache (reusing the `[understand.embed]` backend), and groups rows by semantic similarity.
It requires:
- A **built dataset** (`train/valid/test.jsonl`; run `kibble build` first)
- An **embed backend** configured in `[understand.embed]` with a non-empty `base_url` (e.g. `http://localhost:8090/v1`)
Without either, `kibble cluster` and the auto-run during `kibble build` complete gracefully with a warning; the command and build still succeed (fail-soft).
## Algorithm & determinism
- **Spherical k-means:** k-means variant for cosine distance over unit-normalized vectors (the natural metric for embeddings). Deterministic: uses **k-means++ seeding** with a fixed PRNG seed (`0x00C0_FFEE`) so results are identical across runs, and uses `pts[0]` (the first point) as the initial centroid before random seeding, then runs up to 50 iterations until convergence.
- **Distinctive-term labels:** each cluster's label is formed from its top 3 **distinctive terms** — computed by term-frequency × inverse-cluster-frequency (TF-IDF) using the cluster's tokenized answer texts. A small static **stopword list** (`the`, `and`, `to`, `of`, …) is filtered out first so common grammar particles never dominate a label. This is deterministic, requires no LLM, and picks terms that are frequent within the cluster but rare across others. If a cluster is empty, it falls back to `topic-<c>`.
- **No new dependencies:** all clustering math is hand-rolled using `f32` dot products and normalization.
### LLM-named labels
An optional pass, **gated by `[cluster].llm_labels` (default `false`)**, replaces the distinctive-term labels with human-readable topic names generated by the `[ask]` LLM. When enabled:
- **Each topic is renamed** by sending the LLM its distinctive terms + a sample answer, asking for a short title (2–4 words, Title Case). For example, a `old-css-new-css` cluster becomes `Old vs modern CSS`.
- **One LLM call per clustering run**, using the configured `[ask]` endpoint (`base_url`, `model`, `max_tokens`) and reusing `ASK_API_KEY` / `OPENAI_API_KEY` environment variables. No new endpoint or key required.
- **Applies everywhere `clusters.json` is written** — the standalone `kibble cluster` command, automatic clustering during `kibble build`, and inline rebalancing. This ensures the report output and eval topic breakdown both show the LLM-named labels.
- **Fail-soft:** if the `[ask]` backend is unavailable, the LLM returns an error, the reply is unparseable, or the count differs from the cluster count, the original distinctive-term labels are kept. Clustering and building never break due to an LLM error.
- **Non-reproducible labels:** with `llm_labels` on, the cluster centroids, sizes, and answer assignments remain deterministic across runs, but **label text may vary** run-to-run if the LLM's output changes. Set `llm_labels = false` (the default) for full reproducibility.
```toml
[cluster]
llm_labels = false # default: off; set to true to enable LLM-based topic naming
```
## Command
```bash
kibble cluster # cluster with the default k from [cluster].k (default 12)
kibble cluster --k 8 # cluster into exactly 8 topics
kibble cluster --json # emit JSON instead of a formatted table
```
### Output
#### Table (default)
```
[label-1] 1240 rows · first answer snippet (60 chars max)...
[label-2] 892 rows · another sample text...
...
12 topics over 10000 rows -> clusters.json
```
#### JSON
```json
{
"k": 12,
"topics": [
{ "label": "label-1", "size": 1240, "sample": "first answer..." },
{ "label": "label-2", "size": 892, "sample": "another sample..." }
]
}
```
## Output artifact
Cluster writes `data/clusters.json` (configurable via `[cluster].out`) with full results:
```json
{
"k": 12,
"model": "nomic",
"centroids": [[0.1, -0.05, ...], ...],
"labels": ["label-1", "label-2", ...],
"sizes": [1240, 892, ...],
"samples": ["first answer...", "another sample...", ...]
}
```
- `k` — actual cluster count (clamped to 1 ≤ k ≤ n by kmeans)
- `model` — the embedding model used
- `centroids` — unit-normalized cluster means (the seam for the future balance gate)
- `labels` — distinctive-term labels, one per cluster
- `sizes` — cluster sizes (sum to total answer count)
- `samples` — one sample answer text per cluster (up to 80 chars; the terminal table further truncates to 60 chars for display)
## Configuration
```toml
[cluster]
k = 12 # default
min_cluster_size = 2 # default; 0/1 = off
enabled = true # default
out = "data/clusters.json" # default
llm_labels = false # default: opt-in; set to true for LLM-named topic labels
[understand.embed]
base_url = "http://localhost:8090/v1" # empty (default) = clustering skipped
model = "nomic"
batch_size = 64
store = "data/embeddings"
```
- `[cluster].k` — target number of topics (clamped internally to 1 ≤ k ≤ # rows)
- `[cluster].min_cluster_size` — after k-means, any cluster *smaller than* this (count `< min_cluster_size`) is merged into its nearest surviving centroid (deterministic; `0`/`1` disables merging). At the default `2` this folds only singleton (size-1) topics; raise it to also merge larger runts (e.g. `3` folds size-1/2 clusters). Note: the post-merge topic count can be lower than `k`.
- `[cluster].enabled` — gates the automatic run during `kibble build`; the standalone `kibble cluster` command always runs (subject to dataset and embed-backend checks)
- `[cluster].out` — where `clusters.json` is written
- `[cluster].llm_labels` — when `true`, replaces distinctive-term labels with LLM-generated topic names; default `false` (requires configured `[ask]` endpoint)
- The embedder config (`[understand.embed]`) is **reused**: if the backend is not configured, clustering is skipped with a warning and the build unaffected
- **Note:** `stats.total_documents` and `documents_by_source` (in `stats.json` written by `kibble build`) now count documents from all sources — `data/raw` plus every `[[source]]` section — not just the `data/raw` pile, so a `[[source]]`-only build reports actual document counts.
## Auto-run during build
When `[cluster].enabled = true` AND `[understand.embed].base_url` is set (not empty), `kibble build` automatically runs clustering at the end:
```
Build complete: 1000 docs -> train 780 / valid 110 / test 110 ...
Clustered 1000 rows into 12 topics -> data/clusters.json
```
This is **fail-soft:** if embedding or clustering fails, a warning is printed and the build succeeds anyway. No embed backend? Build still completes and writes the dataset — clustering is simply skipped. This design avoids blocking the core pipeline.
## Rebalancing
**When `[cluster].rebalance = true`**, `kibble build` **clusters the curated training set in-memory, caps over-represented topics** against those centroids, and writes them as `data/clusters.json` so the eval gate and the build cap converge in a single build (fixes #34).
Rebalancing runs **train-only**, over the **full written train** (both clean rows and raw/code rows; valid and test are untouched). It embeds each written-train row, clusters them via spherical k-means (using `[cluster].k`, deterministic seeding), assigns each row to its nearest centroid, and drops rows from topics exceeding `[cluster].max_topic_share` (default `0.35`) of the training set — specifically, the **farthest-from-centroid** rows, keeping the highest-similarity members. The resulting clustering (train-only centroids and post-cap sizes) is persisted as `clusters.json` so `eval` reads the exact same centroids, ensuring convergence in one build.
### Configuration
```toml
[cluster]
rebalance = false # default: disabled (deletes training data)
max_topic_share = 0.35 # default: cap topics at 35% of train split (build-side)
[understand.embed]
base_url = "http://localhost:8090/v1" # required: embed backend
model = "nomic"
```
- `[cluster].rebalance` — opt-in; defaults to `false` (rebalancing is off).
- `[cluster].max_topic_share` — per-topic cap (as a fraction of train size). A topic with ≥ `cap` rows has the farthest-from-centroid `(count - cap)` rows dropped. The cap is clamped to at least 1 row per topic.
- **Embed backend required** — same as clustering itself; if not configured, rebalancing is skipped with a warning and the build unaffected.
### Cap vs gate
**`[cluster].max_topic_share` (build-time cap, default `0.35`) is distinct from `[eval.thresholds].max_topic_share` (eval-time gate, default `0.60`)**:
- **Build-side cap** — `[cluster].max_topic_share = 0.35` — used by `kibble build` to *reduce* topic imbalance by dropping rows
- **Eval-side gate** — `[eval.thresholds].max_topic_share = 0.60` — used by `kibble eval` to *measure* topic balance and pass/fail the dataset
Set the build cap **below** the eval gate (e.g., cap at 0.35, gate at 0.60) so rebalancing makes the gate pass. See `docs/EVAL.md#topic-balance-max_topic_share` for the eval-side details.
### Single-pass best-effort
Rebalancing is **single-pass and deterministic**: it assigns each row once, drops in one pass, and does not iterate. As a result, the **largest topic's final share can sit marginally above the cap**. This is by design (no iterative guarantee) and not a bug — run `eval` to check the final balance. If the gap is large, lower the cap and rebuild.
### Fail-soft behavior
Rebalancing gracefully skips if:
- `[cluster].rebalance` is false (default)
- the train split is empty
- `[understand.embed].base_url` is empty or unreachable
- an embedding error occurs (backend unreachable, init failure, etc.)
(Because the inline path clusters with the *same* embedding model it just used to embed the
train answers, the model-mismatch and dimension mismatches that the *eval* gate guards against
cannot arise here — the centroids and the vectors are produced together.)
In every skip case the build **completes unaffected** — the full dataset is written and `stats.json` records `dropped_topic_rebalanced: 0`. The quieter skips (rebalance off, empty train, empty `base_url`) are silent; an embed failure prints a one-line note to **stderr** — `kibble: rebalance skipped (embed init failed): …` or `kibble: rebalance skipped (embed failed): …` (and, for an unexpected error, `kibble: rebalance skipped: …`).
### Stats reporting
The build summary and `stats.json` report `dropped_topic_rebalanced` — the count of rows dropped by rebalancing.
**Important:** per-source `train` counts in the sources table (within `stats.json`) are **pre-rebalance** — they reflect how many rows each source contributed *before* rebalancing. The top-level `train_examples` total is **post-rebalance** (the final written count after drops). This avoids confusion: you can see which source had skewed topics by looking at the per-source train counts, and you can see how many rows were lost to rebalancing by comparing the top-level total to the sum of per-source counts.
## Follow-ups
- **Issue #25 (stage C – shipped):** `eval` now gates on topic balance via the `max_topic_share` metric — reads centroids from `clusters.json` and assigns each train answer to its nearest topic, then checks if the largest topic exceeds the threshold. See `docs/EVAL.md` for full details, config, and behavior.
- **Issue #26 (shipped):** LLM-named labels via `[cluster].llm_labels` — see the "LLM-named labels" section above.
- **Spec B (shipped):** the same answer-cluster space powers catalog auto-topics when `[classify].enabled = true`. See `docs/CATALOG.md#auto-topics-classify` for details.