kibble 0.1.0

chew through any source into clean datasets — a fast ingestion, RAG & fine-tuning toolkit
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134

# Document catalog — `data/catalog/`

`kibble build` writes a **catalog** of the documents it scans — `data/catalog/documents.jsonl`
(one row per document) and `data/catalog/summary.json` (role/bucket/topic rollups) — classified
against `data/catalog/taxonomy.yaml` and `data/catalog/overrides.json`. The catalog is
**reporting-only**: it inventories what `build` saw, but it is not read by `index`/`ask` (see
"Reporting-only" below).

## What gets cataloged

`documents.jsonl` covers **every source**, not just the local `data/raw` pile — one entry per
document `doc_id`:

- **Files, Web, Blog** — each source builds a `Document` directly (one document per file, per
  fetched page, per blog post), which is classified inline the same way `data/raw` documents are.
- **Dataset, Codebase** — these sources load pre-chunked training **rows**, not `Document`s. Their
  rows are grouped by `doc_id` and **synthesized** into one catalog entry per distinct `doc_id`: the
  entry's `title` is the `doc_id`, and the classification text is the concatenation of that group's
  non-`system` message content (`user`/`assistant` turns), in first-seen order. What a `doc_id`
  represents differs by source:
  - **Dataset** rows that share an explicit `id` collapse into one entry (their turns are
    concatenated) — so one dataset "document" = one entry.
  - **Codebase** rows are keyed **per chunk** (`name:rel/path#i`, e.g. `mycode:lib.rs#0`), so each
    chunk is its own `doc_id` and gets its own entry. A file split into N chunks yields N catalog
    entries, not one. (Collapsing chunks per file is a possible future refinement.)

Each `documents.jsonl` line is a compact JSON object:

```json
{"doc_id":"mycode:lib.rs#0","source":"mycode","title":"mycode:lib.rs#0","role":"code","topics":[],"lora_bucket":"code","rag":true,"is_reference":false,"chars":88}
```

`summary.json` aggregates the same entries:

```json
{
  "documents": 42,
  "by_role": { "style": 30, "code": 10, "reference": 2 },
  "by_lora_bucket": { "style": 30, "code": 10, "reference": 2 },
  "by_topic": { "security": 5, "css": 3 }
}
```

## Classification

Every document is classified by `catalog::classify` (Files/Web/Blog/`data/raw`) or
`catalog::classify_with_fallback` (Dataset/Codebase, via `synth_entries_from_rows`) against the
taxonomy and overrides:

1. **`source_defaults`** (`data/catalog/taxonomy.yaml`) — if the taxonomy has a `source_defaults`
   entry for the document's source, its `role` and starting `topics` are used.
2. **Fallback role** — if there's no `source_default` for the source, Dataset sources fall back to
   no role hint (`None`) and use the usual `is_reference`-based default (`reference`/`style`);
   **Codebase sources default to `role: "code"`** (passed as `Some("code")`). This fallback never
   overrides a configured `source_default` — it only fills in when the taxonomy has nothing for
   that source.
3. **Keyword topics** — `taxonomy.topics` keyword lists are scanned against the title + first 8000
   characters of text (case-insensitively) and any matching topic names are added.
4. **Per-doc overrides** (`data/catalog/overrides.json`, keyed by `doc_id`) — `role`, `topics`,
   `lora_bucket`, and `rag` here win over everything above.

So the precedence, highest to lowest, is: **per-doc override > `source_defaults` > Codebase's
`code` fallback > `is_reference` default**.

## Auto-topics (`[classify]`)

When `[classify].enabled = true`, each catalog document gets a **hierarchical `auto_topic`** (a path
like `Systems › Linux › Memory`, separator `" › "`) and **`topic_confidence`** (the share of that
document's rows assigned to its majority topic). The build also writes `data/catalog/topics.json`,
containing `path`, `size` (document count per topic), and `samples` (up to 3 sample titles per topic).

### How it works

Auto-topics reuse the **train answer-cluster space** — the same centroids as `clusters.json` and
`[cluster].rebalance`. When rebalancing is enabled, that in-memory clustering is reused exactly, so the
catalog's topic space matches `clusters.json`; when rebalancing is off, topics are clustered fresh from
the training split. Each cluster is then LLM-named hierarchically via the `[ask]` LLM (fires whenever
`[ask].base_url` is set — independent of `[cluster].llm_labels`, which only affects plain `kibble cluster`).
Each document is assigned the **majority topic of its rows** — if rows from the same document scatter
across clusters, the document inherits the cluster with the most rows; ties break to the **lowest cluster
index** (deterministic regardless of HashMap iteration order).

### Caveats

- **Opt-in, default off:** `[classify].enabled = false` → build runs with no embed or LLM calls, and
  `documents.jsonl` is byte-identical to offline mode.
- **Requires backends:** needs both `[understand.embed].base_url` (embed backend, e.g. `http://localhost:8090/v1`)
  and an `[ask]` LLM endpoint — the LLM produces the hierarchical names, and the pass skips (rather than
  emitting flat term labels) when it can't.
- **Fail-soft:** a missing/erroring embed backend, no `[ask]` LLM (or an unparseable/wrong-count naming
  reply), no rows, or empty centroids → auto-topics are skipped with a warning to stderr; the rest of the
  catalog is written intact and the build succeeds (never fails).
- **Non-deterministic names:** the hierarchical `auto_topic` paths come from the LLM, so they may vary
  run-to-run. Centroids and document→cluster assignments stay deterministic; only the label text changes.
- **Reporting-only:** the catalog is never read by `index`/`ask` (retrieval runs off `[index].sources`), so
  auto-topics are informational — they do not affect retrieval or downstream tools.
- **Zero-row documents:** if a document produced no rows in the splits, it receives no `auto_topic`
  and `topic_confidence` stays `None`.

## Reporting-only

`kibble index`/`ask` retrieve from `[index].sources` (`data/index/chunks.jsonl`), built and read
independently of the catalog — the catalog's `rag` field is informational (it reflects what a role
*would* imply for RAG-eligibility) and does not gate or filter what `index` actually indexes.
Changing the taxonomy or overrides changes the catalog's inventory, not what gets retrieved.

## Related config

```toml
# data/catalog/taxonomy.yaml
topics:
  security:
    keywords: ["cve", "ransomware"]
source_defaults:
  mycode:
    role: knowledge
    topics: []

# data/catalog/overrides.json
{
  "twitter:12345": { "role": "code", "topics": ["rust"], "rag": false }
}
```

- `data/catalog/taxonomy.yaml` — optional; missing/unparsable file falls back to an empty taxonomy
  (no topics, no source defaults) rather than failing the build.
- `data/catalog/overrides.json` — optional; missing/unparsable file falls back to no overrides.

## See also

- `docs/RETRIEVE.md` — `[index].sources`/`chunks.jsonl`, the actual retrieval path.
- `docs/CLUSTER.md` — `stats.json`'s `total_documents`/`documents_by_source` counts (distinct from
  the catalog: they count documents that produced at least one training row, whereas the catalog
  lists every document scanned, including ones dropped before producing a row).