kibble 0.1.0

chew through any source into clean datasets — a fast ingestion, RAG & fine-tuning toolkit
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140

# Curriculum generation — `kibble tune`

`kibble tune` generates the Unsloth trainer's **sprint curriculum** — phased training datasets organized by source — from the built dataset. It maps each `[[tune.phase]]` to a set of sources, groups rows by source, and writes `sprint/<phase>/train.jsonl` datasets + a `config.yaml` that the trainer reads via `yaml.safe_load`.

## What it tunes on

Tune operates on the **built dataset's training split**: after `kibble build` writes `train.jsonl`, tune reads both `train.jsonl` (the messages) **and** `train.sources.jsonl` (the per-row provenance sidecar that `build` writes), groups rows by source, and assembles them into phases.

It requires:
- A **built dataset** (`train.jsonl` + `train.sources.jsonl`; run `kibble build` first)
- A **`[tune]` configuration** in `kibble.toml` defining the model, LoRA params, phases, and output location

Without either, `kibble tune` fails with a clear message (fail-soft).

## Provenance sidecar

Starting with the `build` refactor, each dataset split (train/valid/test) gets a **provenance sidecar** alongside its `.jsonl` file:

- **`train.jsonl`** — messages-only, one JSON object per line (unchanged)
- **`train.sources.jsonl`** — one source name per line, in the same order as `train.jsonl` rows

Each row's source is the `[source]` name or the derived name of the ingested document (e.g., `twitter-archive`, `readme`, `github-ml-papers`). The sidecar is deterministic (source order matches row order) and immutable after `build` writes it.

When `build` reruns with rebalancing enabled, the sidecar is rewritten to match the new row order after drops, so `tune` always reads matching row/source pairs.

## Configuration

```toml
[tune]
model = "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit"
r = 16                    # LoRA rank (default)
lora_alpha = 16           # LoRA alpha (default)
lora_dropout = 0.05       # LoRA dropout (default)
max_seq_length = 4096     # context window (default)
learning_rate = 2.0e-4    # training LR (default)
smoke_rows = 20           # smoke-test dataset size (default)
out = "data/tune"         # output directory (default)

[[tune.phase]]
name = "phase0"
sources = ["readme", "github-ml"]
max_steps = 100
# resume_from omitted → train this phase from the base model

[[tune.phase]]
name = "phase1"
sources = ["twitter-archive", "blog-posts"]
max_steps = 200
resume_from = "phase0"

[[tune.phase]]
name = "phase2"
sources = ["github-ml"]
max_steps = 150
resume_from = "phase1"
```

- `model` — HuggingFace model identifier (e.g., `unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit`)
- `r` — LoRA rank (intrinsic dimensionality; default `16`)
- `lora_alpha` — LoRA scaling factor (default `16`)
- `lora_dropout` — dropout in LoRA modules (default `0.05`)
- `max_seq_length` — max tokens per training example (default `4096`)
- `learning_rate` — optimizer learning rate (default `2.0e-4`)
- `smoke_rows` — size of the smoke-test dataset for quick iteration (default `20`)
- `out` — directory where sprint curriculum and config are written (default `data/tune`)
- `[[tune.phase]]` — array of training phases:
  - `name` — phase identifier (e.g., `phase0`, `phase1`)
  - `sources` — list of source names to include in this phase (empty list = skip phase)
  - `max_steps` — maximum training steps for this phase (passed to the trainer)
  - `resume_from` — optional phase name to resume adapter weights from; **omit the key** to train this phase from the base model (TOML has no `null` literal — leave `resume_from` out rather than setting it to `null`)

## Output layout

Tune writes to `<out>/` (default `data/tune/`):

```
data/tune/
  sprint/
    phase0/
      train.jsonl          # rows from sources ["readme", "github-ml"]
    phase1/
      train.jsonl          # rows from sources ["twitter-archive", "blog-posts"]
    phase2/
      train.jsonl          # rows from sources ["github-ml"]
    phase0-smoke/
      train.jsonl          # first smoke_rows (20) of the FIRST non-empty phase (dir always named phase0-smoke)
  config.yaml              # trainer config: JSON-formatted (valid YAML)
```

- **`sprint/<phase>/train.jsonl`** — one per non-empty phase, messages-only (verbatim from the built dataset, no re-encoding)
- **`phase0-smoke/train.jsonl`** — smoke test: first `smoke_rows` of the first non-empty phase (for quick trainer iteration)
- **`config.yaml`** — trainer configuration written as **pretty JSON** (indented, sorted keys). Valid YAML syntax for the trainer's `yaml.safe_load` call. Contains model name, LoRA params, max sequence length, batch size, gradient accumulation, learning rate, and the phase specs (name, path, output dir, max steps, resume-from).

## The pipeline

```
kibble build
  ↓
  train.jsonl + train.sources.jsonl
  ↓
kibble tune
  ↓
  sprint/<phase>/train.jsonl + config.yaml
  ↓
kibble pack
  ↓
  Kaggle dataset bundle
  ↓
Upload to Kaggle
  ↓
Run trainer (Unsloth / MLX)
  ↓
Adapter weights + eval
```

- **`build`** reads configured sources and produces `train.jsonl` + `train.sources.jsonl` (per-row source attribution)
- **`tune`** groups rows by source into phases and writes the sprint curriculum
- **`pack`** bundles the curriculum and training scripts for Kaggle
- **Training** (external) reads `sprint/<phase>/train.jsonl` and `config.yaml`, runs the phased curriculum, and outputs adapter weights

## Validation and fail-soft

Tune performs several validation checks and handles errors gracefully:

- **No `[tune]` table:** error with a message pointing to `docs/TUNE.md`
- **Missing `train.jsonl`:** error message suggests running `kibble build` first
- **Missing `train.sources.jsonl`:** error message suggests running `kibble build` first
- **Row/source count mismatch:** error message suggests re-running `kibble build`
- **Unknown source:** any source name referenced by a `[[tune.phase]]` that never appears in `train.sources.jsonl` prints a warning to stderr (`kibble: tune — source '<src>' has no rows in the dataset`). This fires per missing source name, independent of whether its phase ends up empty.
- **Empty phase** (none of its sources contribute rows): skipped — no `sprint/<phase>/train.jsonl` is written for it, and it is omitted from `config.yaml`.
- **No matching sources:** error if *every* phase is empty (no rows matched any source).

All validation checks (missing `[tune]`, missing files, count mismatch, no-match) run **before** any files are written, so those failure modes leave nothing on disk. The writes themselves are a plain sequence of `create_dir_all`/`write` calls (not transactional): an I/O error midway (e.g. disk full) can leave earlier phase directories already written before the command errors out.

## Follow-ups

- **Phase B (deferred):** Training orchestration — automatically run the trainer against the curriculum on Kaggle/MLX, poll for completion, and return adapter weights.
- **Phase C (deferred):** Adapter-back + eval-after-train — pull trained adapters back to the repo, run eval against the fine-tuned model, and report quality metrics.

See `.superpowers/sdd/task-*.md` for the roadmap.