# captchaforge training pipeline
Tools for building + fine-tuning the captchaforge captcha-specific
vision-language model. The Rust side ships:
- `captchaforge::vlm_dataset::VlmDataset` — content-addressed sample
storage with manifest + dedupe.
- `captchaforge::training_corpus::TrainingCorpus` — append-only
failure capture from the production chain.
This directory provides the **Python compute side** — the training
scripts that consume those datasets and produce model weights ready
for `captchaforge::solver::VlmCaptchaSolver` to load via Ollama or
the OpenAI-compatible API.
## Why a separate pipeline directory?
The training side needs CUDA + PyTorch + Transformers + PEFT — heavy
deps that have no business in the production captchaforge consumer's
dependency tree. Operators run training on dedicated GPU boxes
(typically once a week, then promote the new weights into the
solver's Ollama model registry).
The Rust crate stays pure-Rust + chromium. The Python pipeline lives
here, runs offline, ships weights as artefacts.
## Pipeline stages
1. **Collect** — production chain runs with `with_training_corpus()`
set; every failure auto-appends to per-vendor JSONL files. After
~1 week of production traffic you typically have 10k-100k
labelled-by-failure samples per high-volume vendor.
2. **Curate** — turn the failure corpus into a labelled VLM
dataset. Either:
- Hand-label via the operator UI (ships separately).
- Auto-label with a powerful teacher model (GPT-4V / Claude
Opus 4.7 vision) — `scripts/auto_label.py` does this.
3. **Train** — LoRA fine-tune `qwen2-vl-7b` (or any compatible
VLM) on the curated dataset. `scripts/train_lora.py` does this.
Single-GPU recipe runs in ~6h on a 24GB card; multi-GPU recipe
runs in ~90min on 4×A100s.
4. **Distill** (optional) — train a 0.5B-param student model on
teacher labels for CPU-only inference. `scripts/distill.py`.
Distilled model fits in <500MB on disk, runs ~100ms per
inference on a modern CPU. Use as the default for the 99% of
captchas the bigger model isn't needed for.
5. **Promote** — convert the trained weights to GGUF format,
register with Ollama, point `captchaforge`'s VLM solver at the
new model name. `scripts/promote_to_ollama.sh` automates this.
6. **Re-bench** — kick the captchaforge bench against the new
model and confirm the per-vendor accuracy delta is positive
before flipping production traffic. `scripts/bench_compare.py`
diffs two bench runs and refuses promotion when accuracy
regressed on any vendor by >2pp.
## Quickstart
```bash
# Build the training image (one-time, ~15 min on a fast machine).
docker build -t captchaforge-trainer training/
# Mount the production corpus + run an end-to-end fine-tune.
docker run --gpus all \
-v /var/lib/captchaforge/corpus:/work/corpus:ro \
-v /var/lib/captchaforge/datasets:/work/datasets \
-v /var/lib/captchaforge/weights:/work/weights \
captchaforge-trainer \
/work/scripts/train_lora.py \
--base-model qwen/qwen2-vl-7b \
--corpus /work/corpus \
--dataset /work/datasets/round-N \
--out /work/weights/captchaforge-vlm-round-N \
--epochs 3 \
--batch-size 4
```
## Directory layout
```
training/
├── README.md — this file
├── Dockerfile — CUDA + PyTorch + Transformers + PEFT
├── requirements.txt — pinned Python deps
└── scripts/
├── auto_label.py — teacher-model auto-labelling
├── train_lora.py — LoRA fine-tune of a VLM
├── distill.py — student-model distillation
├── promote_to_ollama.sh — convert + register weights with Ollama
└── bench_compare.py — diff two captchaforge bench runs
```
## Ground rules
- **Do NOT pull this directory into a captchaforge consumer's
dependency tree.** The Rust crate is intentionally pure-Rust;
this exists for offline training only.
- **Do NOT commit weights.** The `weights/` directory is
`.gitignore`'d. Use HuggingFace / Ollama / S3 for artefact
storage; commit only the recipes that produced them.
- **Do NOT auto-label production failures with a teacher model
in any case where the vendor's TOS forbids automated solving.**
The auto-label step is for OPEN-source captcha challenges +
internally-controlled fixtures. Per-vendor opt-in only.