rusting_brain 5.0.0

A Rust deep-learning library for training and running transformer language models, dense and mixture-of-experts, on CPU or CUDA.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
# 14. Tokens and a Transformer Language Model

**You need:** chapters 1–10. Chapter 11 if you want it to be fast.

**Time:** 2 hours.

**Full code:** [`code/14_language_model.rs`](code/14_language_model.rs)

Everything so far predicted one thing from a fixed set of features: a price
from four columns, a class from three measurements. A language model predicts
the next piece of text from all the text before it, and then does it again with
its own output appended.

That is the whole idea. Everything else in this chapter is machinery for making
it work at a scale where it is useful.

```
"fn main() {"  →  model  →  "\n"
"fn main() {\n"  →  model  →  "    "
"fn main() {\n    "  →  model  →  "println"
```

---

## 14.1 Next-token prediction is still classification

A language model is a classifier with a very large number of classes. Chapter 6
built a classifier over three species of flower; this one is a classifier over
32,000 possible next tokens, run once per position in the text.

Everything you learned there still applies. Softmax over the outputs. Cross
entropy against the correct answer. The same gradient.

Two things are genuinely new:

- **The input is a sequence, and order matters.** `let x = y` and `let y = x`
  contain the same tokens and mean different things.
- **Every position is a training example.** A 512-token sequence is not one
  training row; it is 511 of them — predict token 2 from token 1, token 3 from
  tokens 1–2, and so on. This is why language models learn from so much less
  human labelling than everything else: the label is the next token, and the
  text labels itself.

The second point is also why the architecture has to be *causal*. Position 5
must not see position 6, or predicting token 6 becomes a lookup rather than a
prediction, and the model learns nothing while reporting a wonderful loss.

---

## 14.2 Tokens

Models work on numbers, so text has to become numbers first. There are three
ways to do it and only one of them is used in practice.

**Characters.** Vocabulary of about 100. Simple, and every string encodes.
But `println` becomes seven separate decisions, sequences get very long, and
attention cost grows with the square of sequence length. Expensive.

**Words.** Vocabulary of hundreds of thousands, and it still cannot spell
`unwrap_or_else` if that word was not in the training data. Unbounded and
brittle.

**Subwords (BPE).** The compromise everything uses. Start from bytes, then
repeatedly merge the most frequent adjacent pair into a new token. Common
things become single tokens — `fn`, `impl`, `String`, `-> Result<` — and rare
things fall back to pieces. Fixed vocabulary, nothing is unrepresentable.

```
"fn main() {"  →  [1023, 887, 12, 13, 45]
```

RustingBrain reads the Hugging Face `tokenizer.json` format, which is how
GPT-2, LLaMA, Mistral, Qwen and most other published models ship their
vocabulary. `Bpe` covers byte-pair files and `Unigram` covers the T5 kind:

```rust
use rusting_brain::Bpe;

let tokenizer = Bpe::from_file("tokenizer.json")?;
let ids: Vec<u32> = tokenizer.encode("fn main() {")?;
let text = tokenizer.decode(&ids);
```

It does not train a vocabulary. To make one for your own corpus, train it with
the [`tokenizers`](https://crates.io/crates/tokenizers) library (Rust or
Python) and load the `tokenizer.json` it writes.

For this chapter's code we will use a character-level tokenizer built in
fifteen lines, so the example runs with no extra dependency and you can read
every part of it. Swap in BPE when you move to real text; nothing else in the
model changes.

The model's only requirement is that every id is less than `vocab_size`. Break
that and you get:

```text
token id 41023 is outside the vocabulary of 32000
```

which almost always means the tokenizer and the checkpoint disagree.

---

## 14.3 Building the model

```rust
use rusting_brain::{Optimizer, TransformerLm};

let mut model = TransformerLm::builder()
    .vocab_size(32_000)
    .d_model(512)
    .n_layers(8)
    .heads(8, 2, 64)
    .d_ff(1408)
    .moe_layers([])            // dense; chapter 15 turns this on
    .max_seq_len(1024)
    .tie_embeddings(true)
    .optimizer(Optimizer::adam(3e-4))
    .seed(42)
    .build()?;

println!("{}", model.parameter_counts());
```

Each knob, and what it costs you:

| Knob | What it is | Effect |
|---|---|---|
| `vocab_size` | Number of distinct tokens | Sets embedding size: `vocab_size × d_model` parameters, often the largest single matrix |
| `d_model` | Width of the vector carrying each token | The main capacity dial. Cost grows with its square |
| `n_layers` | How many blocks stack | The main depth dial. Cost grows linearly |
| `heads(q, kv, dim)` | Query heads, key/value heads, size of each | See below |
| `d_ff` | Hidden width inside the feed-forward | Usually 2.5–4× `d_model`. Most of the parameters live here |
| `max_seq_len` | Longest context ever supported | Baked into the checkpoint. Raising it later means retraining |
| `rope_base` | Rotary frequency base | 10000 is standard. Larger stretches positional resolution over longer contexts |
| `tie_embeddings` | Reuse the embedding matrix as the output projection | Saves `vocab_size × d_model` parameters, costs a little quality |

### Heads, and why there are two counts

Attention runs in parallel "heads", each looking at a different relationship —
one might track matching brackets, another the subject of a sentence. `d_model`
is split across them.

`heads(8, 2, 64)` means eight query heads, two key/value heads, 64 dimensions
each. That is **grouped-query attention**: four query heads share each
key/value head.

Why bother? Because at generation time the keys and values of every previous
token have to be kept in memory (14.6). That cache is the dominant memory cost
of serving a model, and it is proportional to `n_kv_heads`, not `n_heads`. Going
from 8 to 2 cuts it by four with very little quality loss. `n_heads` must be a
multiple of `n_kv_heads`; setting them equal gives you ordinary multi-head
attention.

`head_dim` must be even, because rotary embeddings rotate dimensions in pairs.

---

## 14.4 What one block does

Eight of these, in a row:

```
input ──┬─────────────────────────────┐
        ↓                             │
    RMSNorm                           │
        ↓                             │
    attention (RoPE + causal mask)    │
        ↓                             ↓
        └───────────── + ─────────────┘   residual
                       ↓
                 ──────┬──────────────┐
                       ↓              │
                   RMSNorm            │
                       ↓              │
                   SwiGLU             │
                       ↓              ↓
                       └────── + ─────┘   residual
                              ↓
                           output
```

**RMSNorm** rescales each vector to a consistent magnitude so the next layer
sees inputs in a predictable range. It is LayerNorm without the mean
subtraction — one pass over the row instead of two, and empirically just as
good.

**Attention** lets each position look at earlier positions and pull in what it
needs. This is the part that makes order matter. The *causal mask* is what
stops position 5 seeing position 6.

**RoPE** (rotary position embedding) tells attention where each token is, by
rotating the query and key vectors by an angle proportional to position. The
useful property is that the rotation cancels in the dot product except for the
*difference* in positions, so attention naturally sees relative distance rather
than absolute index.

**SwiGLU** is the feed-forward: `(xW₁ ⊙ swish(xW₂))W₃`. It is where most of the
parameters and most of the arithmetic are, and the multiplicative gate lets the
layer suppress its own channels — which a plain ReLU MLP cannot do.

**Residuals** — the `+` arrows — are what make eight layers trainable at all.
The gradient reaches layer 0 through the additions without passing through
every transformation on the way.

---

## 14.5 Training

```rust
let batch: Vec<Vec<u32>> = vec![
    vec![10, 22, 7, 31, 4],
    vec![9, 18, 44, 2, 61],
];

let loss = model.train_step(&batch)?;
println!("loss {:.4}, perplexity {:.2}", loss.lm_loss, loss.lm_loss.exp());
```

`train_step` does the forward pass, computes the next-token cross entropy,
backpropagates, and updates the weights. One call, one optimizer step.

Sequences in one batch should be the same length. Different lengths are
allowed — shorter ones are padded on the right and the loss ignores the padded
positions — but padding is wasted arithmetic, so chop your corpus into
fixed-length windows instead.

### Read perplexity, not loss

`exp(loss)` is **perplexity**: roughly, how many tokens the model is choosing
between. It is the number worth watching, because it has a meaning. Every loss
the crate returns carries it:

```rust
let loss = model.train_step(&windows)?;
println!("loss {:.4}  ppl {:.2}", loss.lm_loss, loss.perplexity());
```

| Perplexity | What it means |
|---|---|
| = vocab_size | Uniform guessing. An untrained model. |
| 100 | Narrowed 32,000 options to about 100 |
| 20 | Respectable for a small model on narrow text |
| 5 | Very confident; check you are not evaluating on training data |
| 1 | Perfect prediction. On real text this means a bug |

A fresh 32,000-token model starts at a loss near `ln(32000) ≈ 10.4`. If your
first step prints something far from that, the model or the data is wrong
before training has begun.

### TokenBatch, for control

`train_step` builds a `TokenBatch` internally. Build it yourself when you want
padding control or a loss mask:

```rust
use rusting_brain::TokenBatch;

let batch = TokenBatch::new(&sequences)?;
println!("{} sequences × {} tokens", batch.batch(), batch.seq_len());
let loss = model.train_step_batch(&batch)?;
```

`with_loss_mask` marks positions that should not contribute to the loss — the
prompt half of a prompt/answer pair, for instance, when you want the model
graded only on its answer.

---

## 14.6 Generating text

Generation is: run the model, pick a token from the last position's
probabilities, append it, repeat. That loop is written for you:

```rust
use rusting_brain::Sampler;

let mut sampler = Sampler::temperature(0.8, None).top_k(40);
let continuation = model.generate(&prompt_ids, 200, &mut sampler)?;
```

`generate` returns the new tokens only, not the prompt. It builds the KV caches
itself and throws them away when it returns.

### Sampling

The model gives one score per vocabulary entry for the next position. Turning
that row into a token is a separate decision, and `Sampler` holds every knob for
it:

```rust
let mut sampler = Sampler::temperature(0.8, Some(42))
    .top_k(40)
    .top_p(0.95)
    .repetition_penalty(1.1);
```

**Greedy** (`Sampler::greedy()`) — always the highest score. Deterministic, and
it loops: "the the the". Use it when you are measuring something, not reading
it.

**Temperature** — divides the logits by `T` before the softmax. `T < 1` sharpens
toward the top choice, `T > 1` flattens toward uniform. `0.7–0.9` is the usual
range; `T = 0` is greedy. The `seed` argument is `Some` for a reproducible run
and `None` for a different continuation every time.

**Top-k** — keeps the `k` highest and samples among those. The point is to cut
the long tail: individually the 30,000 worst tokens are each nearly impossible,
but together they carry enough probability that one eventually gets picked, and
one wrong token derails everything after it. `k = 40` is a common default; `0`
keeps all of them.

**Top-p** (nucleus) — keeps the smallest set of tokens whose probabilities reach
`p`, so the number kept follows how confident the model is: a handful where the
model is sure, hundreds where it is not. `0.95` is a common default; `1.0` keeps
all of them.

**Repetition penalty** — divides the logit of every token already generated by
`penalty`, which is how a small model is stopped from repeating one phrase for
as long as you let it run. `1.0` leaves the logits alone; `1.1` is gentle.

Temperature reshapes the distribution and the two truncations cut it. They
compose, and they are applied in that order.

### Tokens as they arrive

A hundred tokens take as long as a hundred forward passes, so a program that
prints the result at the end prints nothing for several seconds.
`generate_with` hands over each token at the moment it exists, which is also
where you notice an end-of-text id or a stop sequence:

```rust
let continuation = model.generate_with(&prompt_ids, 200, &mut sampler, |id| {
    print!("{}", tokenizer.decode(&[id]));
    use std::io::Write;
    std::io::stdout().flush().ok();
    id != end_of_text
})?;
```

Returning `false` stops the generation. The token that stopped it is still in
the returned `Vec` — it was sampled, and hiding it would make `generate` and
`generate_with` disagree about what the model produced.

### More than one turn

`generate` drops its caches on return, so a chat loop that calls it once per
turn re-reads the whole conversation every time — quadratic in the number of
turns. A `Decoder` keeps them:

```rust
let mut decoder = model.decoder();

decoder.feed(&prompt_ids)?;                 // the user's first message
for _ in 0..100 {
    let id = decoder.next(&mut sampler)?;
    if id == end_of_text { break; }
}

decoder.feed(&next_message_ids)?;           // the second turn
let id = decoder.next(&mut sampler)?;       // nothing above is read again
```

`feed` reads only the ids you hand it. `decoder.history()` is everything the
session has seen, prompts and generated tokens alike — which is what you write
back to the screen or save as the transcript.

### Why any of that is fast

Done naively, generation is quadratic. To produce token 500 you would re-run all
499 previous tokens through all eight layers, and you have already done that
work 499 times.

The **KV cache** is what all three functions above use to avoid it, and if you
want the loop open — a custom stopping rule, a beam search, two models in
lockstep — it is three lines:

```rust
let mut caches = model.new_kv_caches();

// Prefill: the whole prompt in one pass.
let mut logits = model.forward_cached(&prompt_ids, &mut caches)?;

let mut generated = prompt_ids.clone();
for _ in 0..200 {
    let next = sampler.pick(logits.row(logits.rows - 1), &generated);
    generated.push(next);

    // Decode: one token, attending to everything cached.
    logits = model.forward_cached(&[next], &mut caches)?;
}
```

The caches carry the position, so you never track it yourself. Prefill is one
large matrix multiply over the prompt; each decode step is one small one. On
the default model that is roughly 2 ms per token on a CPU instead of 2 ms × the
sequence length.

Cost: memory. The cache holds `2 × n_layers × n_kv_heads × head_dim` floats per
token — which is precisely why `n_kv_heads` is 2 and not 8.

---

## 14.7 Saving

Dense networks used JSON (chapter 9). A transformer should not: JSON costs
about ten bytes per weight, so a 100M-parameter model becomes a gigabyte of
text.

```rust
model.save_bin("model.rbw", Precision::F32)?;   // 4 bytes per weight
model.save_bin("model.rbw", Precision::Q8)?;    // 1 byte per weight, lossy

let model = TransformerLm::load_bin("model.rbw")?;
```

The file carries the architecture as well as the weights, so `load_bin` needs
no builder — it reconstructs the model it was saved from.

**`F32` for anything you will resume.** `Q8` rounds each weight to 256 levels
with one scale per row, costing around 0.4% per weight. For inference that is
invisible. For a resumed Adam run it is a visible step in the loss curve, and
you will spend an hour looking for a bug that is just rounding.

Resuming also needs the optimizer:

```rust
model.save_optimizer_state("model.rbw.opt")?;
// later
model.load_optimizer_state("model.rbw.opt")?;
```

Without it, Adam's moment estimates restart at zero and the first few hundred
steps take much larger effective steps than they should. Chapter 16.4 covers
the ordering rule that makes this actually work on a GPU.

---

## 14.8 The whole thing, small

[`code/14_language_model.rs`](code/14_language_model.rs) trains a character-level
model on one paragraph until it can reproduce it, then generates from a prompt.
It is deliberately tiny — 0.6M parameters, a few seconds on a CPU — because the
point is to watch the machinery work end to end.

```bash
cargo run --release
```

```text
354 characters, 27 distinct -> vocabulary of 27
19 training windows of 64 tokens
0.6M total / 0.6M active (1.00x)

step    loss   perplexity   (uniform = 27)
   1  3.2626      26.12
 100  0.0165       1.02
 600  0.0124       1.01

prompt: "the borrow checker"
  T=0.2 k=8   "the borrow checker is not your enemy. it is a colleague who has read the code mo thaoref e de cisa"
  T=0.8 k=8   "the borrow checker is not your enemy. it is a colleague who has read the code mo thaoref e de cisa"
  T=1.5 k=27  "the borrow checker is not your enemy. it is a colleague who has read the code mo thaoref e de cisa"
```

Three things in that output are worth stopping on.

**Step 1 perplexity is 26.12 against a vocabulary of 27.** The untrained model
is guessing almost uniformly, exactly as 14.5 said it would. This is the
cheapest sanity check you have, and it catches a wrong `vocab_size` or
out-of-range ids before you waste an hour.

**Perplexity 1.01 means memorization, not language.** The model has 0.6M
parameters and 354 characters of text. It is not learning English; it is
storing the paragraph. That is the chapter 13.9 overfit test — proof the
pipeline works — and nothing more. On real text at this perplexity you have a
bug, almost always training data leaking into evaluation.

**Generation is verbatim for about 64 characters, then falls apart.** That is
`SEQ_LEN`. The model never saw a position past 64 during training, so rotary
embeddings are being asked to extrapolate to angles they were never fit for,
and the output degrades into plausible-looking rubble. Train on longer windows
and the boundary moves. This is the same effect people hit when they prompt a
real model past its context length.

Notice too that all three temperatures produce identical text — even `T=1.5`
with no top-k cut at all. When a model has memorized its corpus the probability
mass is a single spike: the top logit is twenty or more above the next, and
dividing by 1.5 does not close that gap. **Temperature only matters when the
model is genuinely uncertain.** On a real model, trained on more text than it
can store, the same three settings are three visibly different paragraphs.

## 14.9 If something went wrong

| Symptom | Cause |
|---|---|
| `token id N is outside the vocabulary of M` | Tokenizer and model disagree. They must be saved and loaded together |
| `sequence length N exceeds the configured maximum M` | `max_seq_len` is baked in at build time; this means retraining |
| `head_dim must be even for rotary embeddings` | RoPE rotates pairs |
| `n_heads must be a multiple of n_kv_heads` | Grouped-query attention needs whole groups |
| First loss is not near `ln(vocab_size)` | Wrong `vocab_size`, or ids out of range |
| Loss drops to near zero on real text | Evaluating on training data, or your windows overlap |
| Generation repeats one token forever | Greedy sampling. Add temperature and top-k, or a `repetition_penalty` |
| Generation is fluent nonsense | Undertrained. This is what a small model on little data does |
| Generation is `NaN` or one id | Overflow in a hand-written softmax — subtract the max, or use `Sampler` |
| Decoding slows down as it goes | Not using `forward_cached`, or rebuilding the caches each step |

---

## Exercises

1. Print the first `train_step` loss for `vocab_size` 100, 1000, and 32000.
   Confirm each is near `ln(vocab_size)`.
2. Generate the same prompt at temperature 0.1, 0.8, and 1.5. Then at
   `top_k` 1, 40, and `vocab_size`. Then at `top_p` 0.5 and 0.95, and explain
   why the second one changes less than you expected on a certain model.
3. Time 100 tokens with `forward_cached` against 100 tokens re-running the
   whole prefix through `forward_train` each step.
4. Set `tie_embeddings(false)` and compare `parameter_counts()` and the loss
   after 200 steps. Was the saving worth it?
5. Replace the character tokenizer with `Bpe::from_file` on a published
   `tokenizer.json`. Note how many fewer tokens the same text becomes.
6. Raise `SEQ_LEN` from 64 to 128 and retrain. Confirm the coherent stretch of
   generated text grows with it, and explain why.

---

## Recap

- A language model is a classifier over the vocabulary, run at every position.
  Every position is a training example, which is why text labels itself.
- The causal mask is what keeps it a prediction rather than a lookup.
- Subword (BPE) tokenization is the practical middle between characters and
  words. `Bpe` reads a published vocabulary; training one is left to the
  `tokenizers` library.
- `d_model` and `n_layers` are the capacity dials; `d_ff` holds most of the
  parameters; `max_seq_len` is baked into the checkpoint.
- Grouped-query attention exists to shrink the KV cache, which is the dominant
  memory cost of serving.
- A block is RMSNorm → attention+RoPE → residual → RMSNorm → SwiGLU → residual.
- Watch perplexity, not loss. An untrained model sits at `vocab_size`.
- `generate` is the whole decode loop; `generate_with` hands you each token as
  it arrives; `Decoder` keeps the caches between turns of a conversation.
- `forward_cached` plus a KV cache turns generation from quadratic to linear,
  and is there when you want the loop open.
- `Sampler` holds temperature, top-k, top-p and the repetition penalty — the
  knobs that stop generation looping or derailing.
- `save_bin` with `F32` to resume, `Q8` to ship. JSON is for dense networks.

---

**Next:** [15. Mixture of Experts](15_mixture_of_experts.md)