# Training — `kibble train`
`kibble train` runs your configured trainer command against the `kibble tune` package. It's **backend-agnostic**: you supply an argv list (no shell), and `kibble train` validates the tune package, substitutes placeholders, spawns the command with the right working directory, and exits nonzero if it fails.
## What it does
`kibble train` executes the `[train].command` (an argv list, not a shell string) against the sprint curriculum produced by `kibble tune`. It:
1. **Validates the tune package** — checks that `<tune_out>/config.yaml` exists and at least one `<tune_out>/sprint/<phase>/train.jsonl` is present (i.e. that `kibble tune` has been run)
2. **Resolves placeholders** — substitutes `{config}`, `{out_dir}`, `{sprint_dir}`, `{model}` in all command arguments (absolute paths; unknown tokens error)
3. **Appends extra args** — if you pass `-- <args>`, they are appended and substituted just like the command args
4. **Spawns the command** — runs it with `cwd` defaulting to `[tune].out`, inheriting stdio so output streams live
5. **Exits nonzero on failure** — if the trainer exits nonzero (or fails to spawn), `kibble train` exits `1`; `--dry-run` prints the resolved command + cwd and spawns nothing
## Configuration
The `[train]` section in `kibble.toml`:
```toml
[train]
command = ["python", "scripts/train_kaggle.py", "--config", "{config}"]
# Optional: cwd defaults to [tune].out (usually data/tune)
cwd = "some/other/dir"
```
- **`command`** (required) — argv list (each element is a separate arg; no shell parsing). Examples:
- `["python", "scripts/train_kaggle.py", "--config", "{config}"]` (Unsloth / Kaggle)
- `["mlx_lm.lora", "--config", "{config}"]` (MLX local)
- `["./train.sh", "{out_dir}"]` (generic shell script)
- **`cwd`** (optional) — working directory for the command (repo-relative path). Defaults to `[tune].out` (e.g. `data/tune`)
## Placeholders
All placeholders resolve to **absolute paths** (except `{model}`, which is a string). An unknown `{token}` is a hard error.
| `{config}` | `<tune_out>/config.yaml` | `/home/user/repo/data/tune/config.yaml` |
| `{out_dir}` | `<tune_out>` (the tune package root) | `/home/user/repo/data/tune` |
| `{sprint_dir}` | `<tune_out>/sprint` | `/home/user/repo/data/tune/sprint` |
| `{model}` | `[tune].model` string (NOT a path) | `unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit` |
**Example resolution:**
```toml
[tune]
model = "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit"
out = "data/tune"
[train]
command = ["python", "scripts/train_kaggle.py", "--config", "{config}", "--model", "{model}"]
```
When run from `/home/alice/project/`, resolves to:
```
["python", "scripts/train_kaggle.py", "--config", "/home/alice/project/data/tune/config.yaml", "--model", "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit"]
cwd: /home/alice/project/data/tune
```
## Flags
**`--dry-run`** — Print the resolved command and working directory, then exit (spawn nothing). Useful for verifying placeholders before committing to a long training run.
```bash
kibble train --dry-run
# Output:
# would run: python scripts/train_kaggle.py --config /path/to/data/tune/config.yaml --model model/x
# cwd: /path/to/data/tune
```
**`-- <extra>`** — Append substituted args to the command. Everything after `--` is appended to `[train].command` and placeholder-substituted.
```bash
# If [train].command = ["mlx_lm.lora", "--config", "{config}"]
kibble train -- --steps 500
# Runs: ["mlx_lm.lora", "--config", "/path/to/config.yaml", "--steps", "500"]
```
## Worked examples
### Unsloth / Kaggle
Upload the tune package (from `kibble pack`) to Kaggle, then run the Unsloth trainer. Configure:
```toml
[train]
command = ["python", "scripts/train_kaggle.py", "--config", "{config}"]
```
This passes the absolute path to `config.yaml` to your training script, which reads it with `yaml.safe_load()`.
### MLX (local)
Train locally on a Mac or Apple Silicon box. Configure:
```toml
[train]
command = ["mlx_lm.lora", "--config", "{config}", "--output-dir", "{out_dir}/adapters"]
```
The `mlx_lm.lora` command-line tool reads the YAML config and writes the trained adapter to your specified output dir.
### Generic shell script
Wrap your trainer in a shell script and invoke it:
```toml
[train]
command = ["./train.sh", "{out_dir}", "{model}"]
cwd = "trainer" # run from the trainer/ directory
```
In `trainer/train.sh`, read the arguments:
```bash
#!/bin/bash
OUT_DIR="$1"
MODEL="$2"
python my_trainer.py --base-model "$MODEL" --output "$OUT_DIR/adapter"
```
## The Tune C runbook
This documents the remote/manual end-to-end process (no code — all by hand for now).
### Phase 1: Prepare the package
1. **Build the dataset** — `kibble build` to create `train.jsonl` + `train.sources.jsonl` in `data/datasets/unsloth/`
2. **Generate the curriculum** — `kibble tune` groups rows by source into phases, writing `data/tune/sprint/<phase>/train.jsonl` + `config.yaml`
3. **Package for Kaggle** — `kibble pack` bundles the curriculum + training scripts into `kaggle/dataset.zip` and a staging dir with metadata
### Phase 2: Train (remote)
1. **Upload to Kaggle** — Push `config.yaml` and the dataset ZIP to Kaggle Datasets or create a Kaggle Notebook
2. **Run the trainer** — On the GPU box (Kaggle Notebook or your own cloud box):
- Unzip the dataset
- Run `kibble train` (if you have `kibble` on that box, with the same `kibble.toml` config pointing to the uploaded package), OR
- Run your trainer directly with the same arguments: `python scripts/train_kaggle.py --config /path/to/config.yaml`
3. **Generate the adapter** — The trainer writes the trained LoRA weights (usually `adapter.safetensors` or similar)
### Phase 3: Bring back and evaluate
1. **Download the adapter** — Copy the trained adapter weights back to your repo (e.g. into `data/adapters/phase0/`)
2. **Merge and serve** — Use your framework's merge utility to combine the base model + adapter for serving (e.g. Unsloth's `merge_and_unload()`)
3. **Benchmark the tuned model** — Run `kibble bench` (the `research` method for grounded, tool-using evaluation) with a served endpoint (vLLM, MLX, or Unsloth inference) pointing to your merged model to score quality + speed
4. **Confirm improvement** — Use `kibble ask` to ask questions against the indexed corpus and verify the tuned model answers better than the base
## Validation and fail-soft
`kibble train` validates before spawning:
- **No `[train].command`** — error with examples for unsloth and MLX
- **Incomplete tune package** — a missing `config.yaml` *or* no `sprint/<phase>/train.jsonl` produces the same `tune package not found at <dir> — run 'kibble tune' first` error
- **Unknown placeholder** — error naming the bad `{token}`
If validation fails, no process is spawned.
## No shell, no env expansion
`command` is an **argv list, not a shell string**. Each element is passed as a separate argument:
```toml
# OK: separate args
command = ["python", "-c", "print('hello')"]
# WRONG: shell string (this is passed as a single arg)
command = ["python -c \"print('hello')\""]
# WRONG: env expansion
command = ["python", "$SCRIPT"] # literal "$SCRIPT", not expanded
```
If you need to run a complex command (pipes, redirects, env vars), wrap it in a shell script and invoke that:
```toml
command = ["./run_trainer.sh"]
```
```bash
#!/bin/bash
# run_trainer.sh
python my_trainer.py --config "$1" 2>&1 | tee train.log
```
Then invoke with extra args:
```bash
kibble train -- --steps 500
# Runs: ./run_trainer.sh --steps 500
```
## See also
- `docs/TUNE.md` — how `kibble tune` generates the sprint curriculum and `config.yaml`
- `docs/PACK.md` — how `kibble pack` bundles the curriculum for Kaggle
- `docs/BENCH.md` — benchmarking the trained model after adapter download