kibble 0.1.0

chew through any source into clean datasets — a fast ingestion, RAG & fine-tuning toolkit
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140

# Understand layer — embeddings, semantic dedup & leakage

The understand layer adds a dense-embedding backend to `kibble`, plus two **opt-in** consumers
built on top of it: semantic near-duplicate collapsing in `build` and semantic near-duplicate /
leakage metrics in `eval`. It is off by default and fails soft — an unconfigured or unreachable
embedding endpoint never breaks a `build` or `eval` run.

## Backend (`[understand.embed]`)

`kibble` talks to any OpenAI-compatible `/embeddings` endpoint. The live backend used in
development is `llama-server --embedding` running on localhost
(`http://localhost:8090/v1`, `nomic-embed-text`, 768-dim).

```toml
[understand.embed]
base_url   = "http://localhost:8090/v1"  # empty (default) = disabled
model      = "nomic"                          # default
batch_size = 64                                # default
store      = "data/embeddings"                 # default
```

- `base_url` — empty by default, which disables the understand layer entirely (both consumers
  below become no-ops). Point it at any OpenAI-compatible `/embeddings` endpoint to enable it.
- `model` — the model name sent in the request body, and part of the store's cache key.
- `batch_size` — how many texts are embedded per request.
- `store` — where the vector cache lives (see below).

**Auth:** the API key comes from the `EMBED_API_KEY` environment variable, falling back to
`OPENAI_API_KEY`. Keys are **env-only** — never read from `kibble.toml`. When set, it's sent as a
`Bearer` token; when unset, requests go out with no `authorization` header at all, which is fine
for a self-hosted endpoint on a trusted network (e.g. a local `llama-server`).

```bash
# Embedding endpoint auth for the understand layer (optional on a trusted LAN).
EMBED_API_KEY=
```

## Store

Embeddings are cached in a persistent, model-scoped vector store under `[understand.embed].store`
(default `data/embeddings/`):

- `vectors.f32` — a packed little-endian `f32` blob (all vectors, back to back).
- `index.json` — `{dim, entries: {hash: row}}`, mapping each content hash to its row in the blob.

Each entry is keyed by `sha256(model ‖ 0x00 ‖ text)`, so:

- The same answer text is embedded **once**, no matter how many times it's seen — `build` embeds
  the corpus once, and a later `eval` run reuses those cached vectors instead of re-embedding.
- The store is **model-scoped**: switching `[understand.embed].model` changes the hash for every
  text, so different embedding models never mix vector spaces in the same store.

## Semantic dedup (`[curate].semantic_dedup`)

```toml
[curate]
semantic_dedup     = false  # opt-in
semantic_threshold = 0.90   # cosine similarity (default)
```

When enabled (and `[understand.embed].base_url` is set), `build` embeds every candidate answer
and runs a hand-rolled **SimHash LSH** pass over the vectors: random hyperplanes produce a 64-bit
sign signature per vector, banded into candidate buckets, confirmed by exact cosine similarity,
and merged with union-find. Answers whose cosine similarity is at or above `semantic_threshold`
collapse into one cluster, and only the **longest** answer in each cluster survives — this
catches near-paraphrase duplicates that exact/normalized dedup and even MinHash near-dedup miss
(different wording, same meaning), while keeping the most information-dense version.

The number of rows dropped this way is reported as `dropped_semantic_duplicates` in
`stats.json`, alongside the existing `dropped_duplicates` / `dropped_filtered` /
`dropped_near_duplicates` counters.

This is a **destructive** stage — it changes what ends up in the dataset — so it's off by
default and should be enabled deliberately (see Rollout below).

## Semantic metrics (`[eval].semantic`)

```toml
[eval]
semantic = false  # opt-in

[eval.thresholds]
max_semantic_near_dup_rate = 0.02  # default
max_semantic_leakage_rate  = 0.0   # default
semantic_threshold         = 0.90  # default (cosine)
```

When enabled (and `[understand.embed].base_url` is set), `eval` computes two additional metrics
using the same SimHash machinery:

- **`semantic_near_duplicate_rate`** — the fraction of `train` answers that collapse into a
  near-duplicate cluster at `semantic_threshold` (read-only measurement — nothing is dropped).
- **`semantic_leakage_rate`** — the fraction of `valid`/`test` answers that are a semantic
  near-match (cosine ≥ `semantic_threshold`) of something already in `train` — i.e. leakage that
  exact-match and MinHash leakage checks can't see because the wording differs.

Both metrics are gated against `max_semantic_near_dup_rate` / `max_semantic_leakage_rate` exactly
like every other `[eval.thresholds]` metric: they count toward `quality_score` and can fail the
overall verdict under `--strict`.

`eval` reuses whatever vectors `build` already cached in the store (same model, same content
hash) instead of re-embedding the corpus.

## Rollout

Because semantic dedup is destructive and semantic metrics are read-only, the recommended order
is:

1. Set `[understand.embed].base_url` and enable `[eval].semantic` first. Run `kibble eval` and
   look at `semantic_near_duplicate_rate` / `semantic_leakage_rate` on the real corpus at the
   default threshold (0.90) — this is purely observational, nothing changes.
2. Once the numbers look right (thresholds make sense for the corpus, no surprising leakage),
   enable `[curate].semantic_dedup` in `build` to actually collapse near-paraphrase duplicates.

## Fail-soft behavior

Both consumers are guarded the same way: if `[understand.embed].base_url` is empty (the
default), the semantic stage is skipped entirely — no network call is attempted. If the endpoint
is configured but unreachable or returns an error, the failure is caught, a warning is printed to
stderr (e.g. `kibble: semantic dedup skipped (embed failed): ...`), and `build`/`eval` continue and
complete exactly as they would with the feature off — `dropped_semantic_duplicates` stays `0`,
and `semantic_near_duplicate_rate` / `semantic_leakage_rate` are reported as `null` (absent from
the gated metrics) rather than failing the run.

## Testing

No new crate dependency was added for any of this (the endpoint client reuses the existing
`reqwest`-based `net` client; SimHash is hand-rolled next to the existing MinHash near-dedup). All
tests are deterministic and run offline against a stub embedder (`StubEmbedder`, identical text →
identical vector) or a localhost HTTP mock — no real network calls or embedding models are needed
to exercise the store, SimHash, or the `build`/`eval` wiring.

## Out of scope (follow-ups on the same store)

The vector store and embedder are deliberately generic so the same infrastructure can support,
without changes to the store format:

- `kibble ask` — retrieval-augmented generation (RAG) answer synthesis on top of `kibble search`
  (which now ships — see `docs/RETRIEVE.md`) — plus web-augmentation of retrieval.
- Clustering / auto-classification of documents using the cached embeddings.