kibble 0.1.0

chew through any source into clean datasets — a fast ingestion, RAG & fine-tuning toolkit
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

# Model benchmarks (`kibble bench`)

`kibble bench [--strict] [--stream]` runs a served model (OpenAI-compatible endpoint) against the
configured benchmark suite, scoring quality + latency and writing `bench/report/{report.json,report.md}`.
`--strict` exits nonzero on FAIL so CI can gate a model; `--stream` streams the model's output to
**stderr** during `method="research"` runs for live visibility (research-only; scoring and the stdout
table are byte-identical with or without the flag).

## Config

```toml
[bench]
out = "bench/report"
[bench.model]
base_url = "http://localhost:8000/v1"   # your vLLM / any OpenAI-compatible server
model    = "style-lora"
temperature = 0.0
max_tokens  = 512
# [bench.judge]  base_url/model  — only needed for method = "judge"

[[bench.benchmark]]
name = "knowledge"
path = "benchmarks/knowledge.jsonl"
method = "contains"      # contains | mcq | judge
threshold = 0.7          # min score to pass; optional: min_tokens_per_sec, system_prompt
```

## Methods & case formats
- **contains** — `{prompt, expected:[strings]}`; passes if the output contains any expected substring.
- **mcq** — `{prompt, choices:[...], answer:"B"}`; the letter is appended and parsed back.
- **judge** — `{prompt, rubric}`; a judge model returns `{"score":0..1}` (1–5 auto-normalized).

Add a benchmark = add a `[[bench.benchmark]]` + a `cases.jsonl`. Keys from env
(`BENCH_API_KEY`/`OPENAI_API_KEY`, `JUDGE_API_KEY`).

## Web-research method (`method = "research"`)

Runs a tool-execution loop — the model calls `web_search` / `fetch_page`, the harness executes them,
and the **final** answer is scored (contains/judge) plus **turns / tool-calls / wall-time**.

```toml
[bench.search]
# base_url = "http://localhost:8888"   # SearXNG (must enable `formats: [json]`); unset → DuckDuckGo scrape
max_results = 5
allow_hosts = ["127.0.0.1"]            # SSRF bypass for fetch_page (add trusted hosts)

[[bench.benchmark]]
name = "research"
path = "benchmarks/research.jsonl"
method = "research"
scorer = "contains"     # score the final answer: contains | judge
threshold = 0.6
max_turns = 4
```

- `fetch_page` is **SSRF-guarded** (blocks loopback/private unless in `allow_hosts`) and byte-capped
  (`fetch_bytes`, default 2 MB) + char-truncated (`fetch_chars`, default 4000).
- Report shows `turns` and `wall ms` (end-to-end) alongside model-side `latency ms`.

> **Note:** `tok/s` and `min_tokens_per_sec` are not meaningful for `research` benchmarks (tokens
> arent accumulated across the tool loop) — gate research on `mean_wall_ms`/score instead.