oxidelm 0.2.0

Bare-metal pure-Rust Transformer engine with reverse-mode autograd, BPE tokenization, and wgpu acceleration
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
# OxideLM Glossary

A reference of technical terms used throughout the OxideLM project. Each entry includes a name and a succinct explanation of how the term is used in this codebase.

---

## A

### Activation Function
A non-linear function applied element-wise to tensor values. OxideLM uses **ReLU**, **Sigmoid**, **GELU**, and **SiLU** (the latter in the SwiGLU feed-forward block).

### AdamW
An adaptive stochastic optimisation algorithm that combines **Adam** with decoupled **weight decay**. Maintains per-parameter first and second moment estimates (`beta1`, `beta2`) and applies bias correction and L2 regularisation independently of the adaptive learning rate. Implemented in `src/optimizer.rs` for both CPU and GPU.

### AdamWConfig
The hyperparameter record for the AdamW optimiser: `beta1`, `beta2`, `epsilon`, `weight_decay`, plus parameter-group controls (`no_decay_rms_norm`, `no_decay_embeddings`, `rms_norm_lr_scale`, `embedding_lr_scale`). Stored in checkpoints and serialised with Serde.

### AdamWParams
A `#[repr(C)]` struct (`gpu.rs`) holding all AdamW hyperparameters and the per-step corrections, uploaded as a uniform buffer to the GPU optimiser kernel.

### AdamWState
The persisted optimiser state: a step counter and a vector of `ParameterMoments` (first- and second-moment `Vec<f32>`). Saved in checkpoints and reconstructed when training resumes.

### ArchitecturePreset
Named model-size templates (`llama-tiny`, `llama-micro`, `llama-small`, `gpt-tiny`, `gpt-micro`, `gpt-small`) in `src/model.rs`. Each expands to hidden size, layers, heads, KV heads, intermediate size, context length, bias, and tied-embedding defaults. `*-micro` targets ≤8 GiB FP32 training with activation checkpointing. Used by `--architecture` / `model.architecture` and by `oxidelm --train init`.

### AttentionKvCache
A struct holding the **key** and **value** tensors accumulated from past tokens during autoregressive generation. Each transformer layer has its own optional cache, enabling incremental decoding without recomputing the full history. `truncate` drops rejected speculative draft positions.

### Autograd (Automatic Differentiation)
The reverse-mode automatic differentiation engine in `src/lib.rs`. Every operation records an `Op` variant on its output tensor. Calling `.backward()` on a scalar loss tensor performs a post-order topological traversal over the computation graph, computing gradients through each operation in reverse.

### Autoregressive Decoding
A generation mode where the model produces one token at a time, feeding the newly sampled token back as input for the next step. Implemented via `forward_cached` and `forward_cached_gpu` to reuse the **KV cache**.

---

## B

### B (Batch Size)
The first dimension of input tensors (`[B, S, ...]`). Number of independent sequences processed simultaneously. In training config this is the **micro-batch** width (`trainer.batch_size` / `--batch-size`), not the effective batch seen by the optimizer — see [Effective Batch Size](#effective-batch-size).

### Backward Pass
The process of computing gradients by traversing the computation graph in reverse topological order, starting from a scalar loss and propagating gradients to leaves. Entry point: `Tensor::backward()`.

### BackwardParams
A `#[repr(C)]` uniform-buffer struct carrying shape metadata and scalar values to the GPU backward kernels.

### Batched GEMM
General Matrix Multiply applied to batched tensors. OxideLM supports `[B, H, M, K] × [B, H, K, N] → [B, H, M, N]` as `B * H` independent products in one dispatch. The fused attention path instead combines score calculation, online Softmax, and value aggregation without storing the score matrix.

### BOS (Beginning Of Sequence)
A special token (ID 258) optionally prepended to encoded sequences. Stored in `SpecialTokens`.

### Broadcasting
The NumPy-style trailing-dimension broadcasting for element-wise binary operations. When two tensor shapes differ, dimensions of size 1 are stretched to match the other operand. Implemented in `broadcast_shape` and `broadcast_offset` in `lib.rs`.

### BPE (Byte-Pair Encoding)
A subword tokenisation algorithm that starts from raw bytes and iteratively merges the most frequent adjacent token pair. OxideLM implements **byte-level** BPE, meaning the base vocabulary is the 256 single-byte values.

### BpeTokenizer
The main tokenizer struct (`src/tokenizer.rs`) holding the vocabulary (`vocab`), encoder mapping (`encoder`), merge rules (`merges`), merge priority ranks (`merge_ranks`), and special token IDs. Train via `train_from_bytes*`, `train_from_file*`, `train_from_byte_shards*`, `train_from_shards*`, or `train_from_shard_directory*`.

### BYTE_VOCAB_SIZE
A constant equal to 256 — the number of base single-byte tokens in the **byte-level BPE** vocabulary.

---

## C

### Causal Attention
Attention that prevents a token from attending to future positions. Enforced by adding a causal mask (upper-triangular `-∞` entries) before softmax.

### CausalLM (Causal Language Model)
The top-level model struct (`src/model.rs`): a decoder-only Transformer comprising a token embedding, a stack of `TransformerBlock` layers, a final `RmsNorm`, and an LM head projection. Exposes `forward`, `forward_gpu`, `forward_cached`, training helpers, and generation methods.

### CausalLmConfig
The model hyperparameter record, including model widths, query and key/value head counts, context length, normalization, RoPE, initialization, and embedding tying options.

### Causal Mask
An additive mask applied to attention scores before softmax: positions where the query comes before the key get `-∞`, forcing attention weights to zero after exponentiation. For cached decoding the mask also accounts for already-consumed past tokens.

### Activation Checkpointing
A training memory optimisation that discards intermediate activations inside each `TransformerBlock` during the forward pass and recomputes them during backward. Enabled via `TrainerConfig::activation_checkpointing` / `--activation-checkpointing`. Implemented with `checkpoint` / `checkpoint_gpu` and `Op::Checkpoint`.

### Checkpoint
A serialised snapshot of named model weights and optimizer moments (binary LE `F32` payloads with per-tensor CRC32 checksums), tokenizer state, learning-rate schedule, RNG state, data-loader state, trainer position, best validation loss, and consecutive non-improving validation evaluations. Written atomically (`*.tmp` → validate → rename) in the `OXLMCKP2` binary format; loaded with `load_checkpoint`. Legacy JSON float-array checkpoints (version 1) still load.

### Continuous Batching
Serving scheduler (`ContinuousBatcher` in `src/generate.rs`) that batches active decode requests whose KV positions match, lets finished sequences leave the batch, and admits waiting prompts into free slots. `Generator::generate_batch` is the convenience entry point.

### Corpus
Training text discovery (`src/corpus.rs`): a single `.txt` file, a JSONL file of `{"text":…}` records, or a directory of `.txt` / `.jsonl` files. Multi-file and JSONL inputs are concatenated in sorted path order into the token-cache `source/` directory so validation split and token sharding are deterministic.

### CpuMatmulBackend
An enum (`Reference` or `BlockedParallel`) controlling which CPU matrix multiplication implementation is used globally or per call. The default is `BlockedParallel`.

### Cross-Entropy Loss
The standard categorical loss: for each row of logits the negative log-likelihood of the correct class is computed and averaged across all rows. Implemented in `Tensor::cross_entropy` (CPU) and as a GPU kernel in `index.wgsl`.

---

## D

### D (Hidden Size / Model Dimension)
The feature dimension of the Transformer (`hidden_size`). Every token is represented by a vector of length `D` throughout the model.

### Decoder-Only Transformer
A Transformer architecture without an encoder stack. Input tokens attend only to themselves and previous positions (causal attention). Used by GPT-style language models. OxideLM implements this pattern.

### Data Parallel (CPU)
Training with multiple worker threads, each owning a private model replica (`src/data_parallel.rs`). The micro-batch is split along the batch axis, workers run forward/backward independently, and gradients are mean-reduced onto the primary model before the optimizer step. Graphs stay single-threaded (`!Send`); only weight/gradient `Vec<f32>` buffers cross threads. CPU uses `CpuDataParallelPool`; GPU uses `GpuDataParallelPool` with one `GpuContext` per worker from `GpuContext::create_devices` (distinct adapters when available, else multiple logical devices). Configured by `TrainerConfig::data_parallel_workers` (batch size must be ≥ workers and divisible by the worker count). Does **not** multiply [effective batch size](#effective-batch-size); it only parallelizes each micro-batch of size `batch_size`.

### Device
An enum (`Cpu` | `Gpu`) tracking where a tensor's authoritative data currently resides.

### Dispatch
A single GPU kernel invocation. OxideLM launches compute dispatches via `wgpu::ComputePass::dispatch_workgroups`.

---

## E

### ElementParams
A `#[repr(C)]` uniform-buffer struct carrying shape parameters (`length`, `rows`, `columns`, `epsilon`) to element-wise and normalisation GPU kernels.

### Element-Wise Operation
An operation applied independently to each element of one or two tensors (add, sub, mul, div, pow, relu, sigmoid, gelu, silu). GPU element-wise kernels use 256 threads per workgroup.

### Embedding
A lookup table that maps integer token IDs to dense vectors of size `hidden_size`. Implemented in `src/layers.rs`; the `Embedding::forward` method delegates to `Tensor::embedding`.

### EOS (End Of Sequence)
A special token (ID 259) that marks the end of a sequence. During generation, sampling stops when this token is produced.

### Early Stopping
Training halt triggered after a configurable number of consecutive validation evaluations fail to improve best validation loss by at least `early_stopping_min_delta`. Controlled by `early_stopping_patience` (0 disables). Decisions never use training loss.

### Effective Batch Size
Sequences contributing to one optimizer step: `batch_size × gradient_accumulation_steps`. Logged as `effective_batch_size` in the train / dry-run plan. `data_parallel_workers` shards each micro-batch and does not appear in the product. Peak activation memory tracks micro-batch size (per-replica `batch_size / data_parallel_workers` when workers > 1); raise `gradient_accumulation_steps` to grow the statistical batch under a RAM budget. See also [Gradient Accumulation Steps](#gradient-accumulation-steps) and [Data Parallel (CPU)](#data-parallel-cpu).

### EvaluationMetrics
A struct carrying validation results: step, loss, perplexity, token counts, wall time, whether the model improved, path to the best checkpoint, consecutive non-improving evaluations, and whether early stopping fired.

---

## F

### FeedForward
The SwiGLU feed-forward sub-layer inside a Transformer block (`src/transformer.rs`). Contains `gate_proj`, `up_proj`, and `down_proj` linear projections.

### Frequency Penalty
An additive sampling penalty: each token's logit is reduced by `frequency_penalty * count`, where `count` is how often that token appears in the full history.

### FIRST_MERGE_ID
The first token ID assigned to a learned BPE merge (value 260). IDs 0–255 are raw bytes, 256–259 are special tokens.

### BF16 / FP16 / FP32
Floating-point storage formats. OxideLM's normal tensor path uses FP32. Its mixed-precision matmul path supports native FP16 storage when the adapter exposes `SHADER_F16`, portable packed BF16 storage, and an FP32 fallback. All mixed-precision dot products accumulate and output in FP32.

### Mixed-precision training (AMP)
Optional training mode (`MixedPrecisionConfig` / `--mixed-precision`) that keeps FP32 master weights and AdamW state while rounding working weights to FP16 or BF16. Loss, Softmax, RMSNorm, and gradient-norm reductions stay FP32. A dynamic loss scaler multiplies the loss before backward, unscales gradients before the optimizer, skips the step on overflow, and adjusts the scale.

### FusionParams
A `#[repr(C)]` uniform-buffer struct carrying metadata to fused GPU kernels (e.g., residual + RMSNorm, SwiGLU, masked softmax).

---

## G

### GELU (Gaussian Error Linear Unit)
An activation function: `0.5 * x * (1 + tanh(√(2/π) * (x + 0.044715 * x³)))`. Used as an alternative to ReLU in some Transformer variants.

### Generate
The module (`src/generate.rs`) providing text generation with configurable sampling: temperature, top-k, top-p, repetition / frequency / presence penalties, seeded randomness, continuous batching via `ContinuousBatcher`, and speculative decoding via `SpeculativeGenerator`.

### Gradient
The partial derivative of the scalar loss with respect to a specific tensor element. Leaf tensors store gradients in the `grad` field (CPU `Vec<f32>`) or `gpu_grad` field (GPU buffer).

### Gradient Accumulation
When a tensor is used multiple times in a computation graph, its gradient contributions are summed (accumulated) rather than overwritten. Handled by `accumulate_gradient` in `lib.rs`. Distinct from the trainer knob [Gradient Accumulation Steps](#gradient-accumulation-steps).

### Gradient Accumulation Steps
Trainer knob (`trainer.gradient_accumulation_steps` / `--gradient-accumulation-steps`): number of micro-batches per optimizer step. Each micro-batch runs forward/backward with gradients scaled by `1/steps`; after all steps, AdamW runs once. Multiplies [effective batch size](#effective-batch-size) without increasing micro-batch activation peak. Orthogonal to `data_parallel_workers`, which only shards each micro-batch.

### Gradient Clipping
Scaling down gradients when the global norm exceeds `max_gradient_norm`, preventing exploding gradients. Applied in the training loop.

### Gradient Norm
The L2 norm (Euclidean length) of all parameter gradients concatenated. Reported in training logs and used for gradient clipping.

---

## H

### H (Number of Attention Heads)
The dimension representing the number of independent attention heads. The hidden size `D` must be divisible by `H`, yielding a per-head dimension of `D/H`.

### head_dim
The feature dimension of a single attention head: `hidden_size / num_heads`. Must be even because RoPE operates on adjacent pairs in this dimension.

---

## I

### IndexTensor
An integer-only tensor (`Vec<usize> + shape`) for token IDs and class indices. Unlike `Tensor`, it stores exact integers rather than `f32`, enabling direct indexing into embeddings and cross-entropy targets. GPU execution uses typed `u32` index buffers.

### Initialization
The module (`src/initialization.rs`) providing normal, **Xavier uniform**, and **Kaiming uniform** weight initialisation. Full language models use configurable normal initialization; standalone linear and embedding layers retain Xavier initialization.

### Initializer
A seeded `StdRng` wrapper that produces either normal or Xavier-initialised tensors. Residual-output tensors can apply an extra depth-dependent scale.

---

## K

### Kaiming Initialization (He Initialization)
Weight initialisation that samples from `[-√(6/fan_in), √(6/fan_in)]`. Provided in `initialization.rs` for ReLU-style activations.

### Key (K)
In self-attention, each input token is projected to a key vector that determines which positions attend to it. Projected via `k_proj`.

### KV Cache
Short for **Key-Value Cache**. See **AttentionKvCache**.

---

## L

### Layers
The module (`src/layers.rs`) providing `Linear` and `Embedding` building blocks with both CPU and GPU forward implementations.

### Leaf
An `Op::Leaf` variant marking a tensor that was created directly (not as the result of an operation). Leaf tensors are the targets of gradient accumulation during `backward()`.

### LearningRateSchedule
A cosine decay schedule with linear warmup. During warmup steps the learning rate increases linearly to `base_learning_rate`; afterwards it follows a cosine curve down to `minimum_learning_rate`.

### Linear
A linear (fully-connected) layer: `output = input @ weight`. Supports standard weight storage (`[in_features, out_features]`), transposed weight sharing (used for weight tying), and inference-only int8/int4 weight copies with per-output-channel scales. Implemented for CPU (`forward`) and GPU (`forward_gpu`).

### LM Head
The final linear projection (`lm_head`) that maps the final hidden state to vocabulary-sized logits. When `tie_word_embeddings` is enabled, it shares its weight tensor with the input embedding layer.

### Logits
The raw, unnormalised scores output by the LM head, of shape `[B, S, V]`. Fed into cross-entropy loss during training or softmax for sampling during generation.

---

## M

### Memory Budget
Static FP32 footprint planning in `src/memory.rs`. Reports parameter, activation, KV-cache, staging/readback, and optimizer bytes for a model and batch shape, then refuses configurations that exceed the selected CPU/GPU budget or `max_buffer_size`.

### Matmul (Matrix Multiplication)
The product of two 2D matrices (or batched stacks thereof). CPU: `Tensor::matmul` with a choice of `Reference` (triple loop) or `BlockedParallel` (tiled, Rayon-parallelised) backend. GPU: dispatched via WGSL tiled kernels (`matmul.wgsl`, `matmul_vec4.wgsl`).

### MatmulDimensions
A struct computed during matrix multiplication holding the tensor rank, batch count, rows, shared/inner dimension, and columns.

### MatmulParams
A `#[repr(C)]` uniform-buffer struct carrying `(M, K, N, batch)` to the GPU matmul kernel.

### mmap / memmap2
Memory-mapped file I/O. The tokenizer uses `memmap2` to read training text directly from disk without copying it into a Rust `Vec<u8>`, keeping memory usage proportional to the working token-ID array rather than the corpus size. Multi-file shard training maps each file separately and never concatenates shard bytes.

### Moments (Optimizer)
First moment (moving average of gradients) and second moment (moving average of squared gradients) maintained per parameter by **AdamW**. Stored in `ParameterMoments`.

### MultiHeadAttention
The grouped-query self-attention module (`src/attention.rs`). Projects input into Q/K/V, maps groups of query heads to shared K/V heads, applies RoPE and causal attention, then merges query heads and projects back via `o_proj`.

---

## N

### Normalization
See **RMSNorm**.

### Nucleus Sampling (Top-p Sampling)
A sampling strategy that restricts tokens to the smallest set whose cumulative probability mass exceeds `top_p` (e.g., 0.95), preventing long-tail low-probability tokens from being sampled. Configured in `SamplingConfig`.

---

## O

### Op (Operation)
An enum variant in `src/lib.rs` recording the computation that produced a tensor (e.g., `Add`, `MatMul`, `SoftmaxLast`, `Rope`, `CausalMask`, `Embedding`). Used during `backward()` to dispatch the correct gradient computation.

### Optimizer
The module (`src/optimizer.rs`) providing the `AdamW` struct with `step` (CPU) and `step_gpu` methods, state management, and checkpoint persistence.

---

## P

### Parameter Group
Per-parameter AdamW settings (`weight_decay`, `lr_scale`) derived from tensor names. By default RMSNorm scales and selected embeddings (`tok_embeddings.weight`, `lm_head.weight`) receive zero weight decay; group learning-rate scales multiply the schedule LR. Built via `AdamW::new_grouped` / `from_state_grouped`.

### PAD (Padding Token)
A special token (ID 256) used to fill sequence positions that should be ignored by attention. Stored in `SpecialTokens`.

### ParameterMoments
A struct holding the first- and second-moment `Vec<f32>` for a single parameter in the AdamW optimiser.

### Permute
A tensor operation that reorders dimensions. For example, `permute([0, 2, 1, 3])` on `[B, S, H, D/H]` produces `[B, H, S, D/H]`, used to split and merge attention heads.

### Perplexity
The exponential of the cross-entropy loss: `exp(loss)`. A standard intrinsic evaluation metric for language models; lower perplexity means the model assigns higher probability to the correct tokens.

### Prefill
The initial forward pass that processes the entire input prompt before generating tokens. During prefill the KV cache is populated and attention is computed over the full prefix.

### Presence Penalty
An additive sampling penalty: each token that appears at least once in the history has `presence_penalty` subtracted from its logit.

### Prompt
The input text that conditions generation. Encoded into token IDs and fed as the initial input to `CausalLM::generate`.

---

## Q

### Quality evaluation
Standalone checkpoint metrics from `src/quality.rs` / `--eval`: held-out cross-entropy loss and perplexity on fixed non-overlapping windows, top-1 next-token accuracy on the same windows, and a small greedy-completion task suite (built-in Shakespeare prompts or a JSON `{id, prompt, expected}` list) scored by exact match, prefix match, and leading-byte overlap.

### Quantized Linear Weight
An inference copy of a Linear weight: packed codes plus FP32 scales (and optional zero-points). Supports int8/int4/int3/int2, NF4, and FP8 with per-tensor / per-channel / per-group granularity. Round-trips via OxideLM checkpoints, `--export-quantized` SafeTensors (`{module}.weight.qpacked|.qscales|.qzeros`), and GGUF import/export.

### Quantization schema version
`CheckpointQuantization.schema_version` in `OXLMCKP2` metadata (`QUANTIZATION_SCHEMA_VERSION`, currently **3**). Loaders accept `1..=3` and reject `0` or newer versions with `unsupported quantization schema version N (supported 1..=V)`. Unknown `method` / format / granularity strings are also rejected. v3 adds optional `activation` and `kv_cache` blocks for W8A8/W4A8 and quantized KV.

### Quantization scheme matrix
Shipped weight packs: **int8/int4/int3/int2** (sym/asym), **NF4**, **FP8 E4M3/E5M2**; granularity **tensor / channel / group:{32,64,128}**; scales **absmax|minmax|percentile|mse**. Product methods: RTN, GPTQ, AWQ, SmoothQuant, HQQ, SpQR, QuIP#, AQLM. Inference modes: W8A8, W4A8, W4A16. See README “Scheme matrix” and `perf/baselines/quantize_schemes.json`.

### Round-to-Nearest (RTN)
Default `--method rtn` packer: per-block absmax (or other scale method) + nearest integer (or NF4/FP8 codebook). No calibration. Correctness bounds live in `tests/quant_correctness.rs` (`max_abs_dequant_error` ≤ `dequant_error_bound`).

### Quantization-Aware Training (QAT)
Training path that fake-quantizes Linear weights in the forward pass while using a straight-through estimator so gradients update the FP32/BF16 masters. Configure via trainer `qat` (`"int8"` / `"int4"` or `{ "format", "group_size", "learnable_scales" }`) or `--qat` / `--qat-group-size` / `--qat-learnable-scales`. Learnable scales use an LSQ-style `q * s` path (per-channel). After training, `oxidelm --export-qat` (or `CausalLM::export_qat_packed`) packs masters into inference weights.

### QLoRA
Frozen NF4/int4 base weights plus LoRA/DoRA adapters in higher precision. Set `lora.base_quant` (`nf4` / `int4`) and optional `lora.base_group_size` (default 64), or `--qlora-base` / `--qlora-group-size`. Adapters save/load separately via `--export-adapters` / `--import-adapters`.

### QKV Projection
Quadratic projection: projecting the input simultaneously into query, key, and value spaces. In OxideLM this is done via three separate `Linear` layers (`q_proj`, `k_proj`, `v_proj`). A fused QKV WGSL kernel was benchmarked but rejected as slower than the separate approach.

### Query (Q)
In self-attention, each input token is projected to a query vector used to compute attention scores against all keys. Projected via `q_proj`.

### QuIP#
Incoherence-processed weight quantization (`--method quip`). Applies a Walsh–Hadamard transform, RTN-packs, then folds the inverse back into a native packed Linear.

### GGUF
llama.cpp weight container. `--export-gguf` / `--import-gguf` support `Q4_0`, `Q4_K`, `Q5_K`, `Q6_K`, and `Q8_0`. Native mapping: `Q8_0`→`int8/group:32/sym`, `Q4_0`/`Q4_K`→`int4/group:32/asym`; `Q5_K`/`Q6_K` import as FP32 (no native 5/6-bit pack).

### HQQ
Half-Quadratic Quantization (`--method hqq`). Calibration-light alternating scale/zero-point optimization; requires `--asymmetric`.

### SpQR
Sparse-quantized residual path (`--method spqr`): isolate magnitude outliers, quantize the dense body, re-pack. For high-compression when a PPL budget still holds.

### AQLM
Additive quantization (`--method aqlm`): residual multi-codebook approximation folded into a native packed scheme.

---

## R

### Rayon
A Rust library for data parallelism. OxideLM uses Rayon for parallel pair-frequency counting in BPE training and for distributing output rows in the blocked-parallel CPU matmul.

### ReLU (Rectified Linear Unit)
The activation function `max(0, x)`. Available as `Tensor::relu()`.

### Repetition Penalty
A generation parameter that penalises tokens that have already appeared in the generated sequence by dividing their logits by `repetition_penalty` (typically ~1.1), reducing the likelihood of repetitive output. Optional `repetition_penalty_window` limits the history to the last N tokens.

### Reshape
A tensor operation that changes the shape without changing the number of elements or the row-major memory layout. Used extensively to flatten batches or reshape attention heads.

### Residual Connection
A path that adds the input of a sub-layer to its output (`x = x + sublayer(x)`). OxideLM Transformer blocks use two residual connections: one around attention and one around the feed-forward network.

### Reverse-Mode Differentiation
A form of automatic differentiation that computes the gradient of a scalar output with respect to all inputs in a single backward traversal. OxideLM implements this via topological sorting of the computation graph and per-operation gradient rules.

### RMSNorm (Root Mean Square Normalization)
A layer normalisation variant that normalises by the root mean square of activations (rather than variance), then applies a learnable per-feature scale. Used before attention and feed-forward sub-layers. Implemented in `src/norm.rs` with both CPU (`forward`) and GPU (`gpu_rms_norm`) paths.

### RoPE (Rotary Position Embeddings)
A positional encoding method that applies a rotation matrix to adjacent pairs of features in query and key vectors based on their absolute position. The rotation frequency (`theta`) controls the base wavelength. Implemented in `Tensor::rope` (CPU) and `gpu_rope_with_offset` (GPU).

### Row-Major
The memory layout used for all tensors: the last dimension varies fastest in the flat `Vec<f32>` buffer. `strides` are computed to support multi-dimensional indexing.

---

## S

### S (Sequence Length)
The second dimension of input tensors (`[B, S, ...]`), representing the number of tokens in each batch element.

### SamplingConfig
A struct holding generation decoding parameters: `max_new_tokens`, `temperature`, `top_k`, `top_p`, `repetition_penalty`, `repetition_penalty_window`, `frequency_penalty`, `presence_penalty`, `seed`, `return_logprobs`, and `top_logprobs`. Speculative decoding adds a separate `SpeculativeConfig` (`num_draft_tokens`).

### Scaled Dot-Product Attention
The core attention mechanism: `softmax(Q @ K^T / √d) @ V`, where `d` is the per-head dimension. OxideLM evaluates this in K/V tiles with an online Softmax, so it stores one log-sum-exp value per query instead of full score and probability matrices.

### Seed
A `u64` value used to initialise the random number generator for deterministic weight initialisation and text generation.

### SGD (Stochastic Gradient Descent)
The simplest optimisation rule: `value -= learning_rate * gradient`. Available as `Tensor::sgd_step()`.

### Sigmoid
The activation function `1 / (1 + exp(-x))`. Available as `Tensor::sigmoid()`.

### SiLU (Sigmoid Linear Unit / Swish)
The activation function `x * sigmoid(x)`. Used as the gating function inside SwiGLU. Available as `Tensor::silu()`.

### Softmax
The function `softmax(x_i) = exp(x_i - max(x)) / Σ_j exp(x_j - max(x))`. Implemented for the last dimension (`softmax_last_dim`) with numerical stability via max subtraction.

### SpecialTokens
A struct containing the token IDs for `pad`, `unk`, `bos`, and `eos`. Defaults to 256–259.

### Speculative Decoding
Draft-then-verify generation (`SpeculativeGenerator` in `src/generate.rs`). A smaller draft model proposes up to γ tokens (`SpeculativeConfig::num_draft_tokens`); the target verifies them in one multi-token cached forward, accepts a prefix (greedy match or Leviathan rejection sampling), truncates rejected KV positions, and samples a correction or bonus token from the target distribution. Requires matching vocabulary and EOS IDs.

### Staging Buffer
A CPU-accessible GPU buffer used for reading back results. OxideLM maintains a pool of staging buffers (`staging_pool`) to amortise allocation costs across readbacks.

### Storage Buffer
A GPU buffer holding tensor data. OxideLM uses `wgpu::BufferUsages::STORAGE` buffers with `TrackedStorageBuffer` wrappers that decrement global memory counters on drop.

### Strides
A vector computed alongside each tensor's shape that records the product of all later dimensions. Used for converting multi-dimensional indices to a flat buffer offset and for broadcasting logic.

### SwiGLU
A gated activation unit used in the feed-forward sub-layer: `(x @ gate_proj) · silu(x @ gate_proj) ⊙ (x @ up_proj)`, then projected back via `down_proj`. The GPU path uses a fused `gpu_swiglu` kernel that is 23% faster than separate dispatch.

---

## T

### Temperature
A generation parameter that controls output randomness. Logits are divided by `temperature` before softmax: `temperature → 0` yields greedy (deterministic) sampling; higher values increase diversity.

### Tensor
The core data structure (`src/lib.rs`): an `Rc<RefCell<TensorData>>` holding a `Vec<f32>` (CPU) or `GpuTensor` (GPU), shape, strides, optional gradient, device marker, and the operation that produced it. The autograd graph is intentionally single-threaded (`!Send` / `!Sync`); Rayon and GPU parallelism apply only to kernels and host helpers that use plain buffers, not shared graph nodes.

### Tie Word Embeddings
When `CausalLmConfig::tie_word_embeddings` is `true`, the token embedding matrix and the LM head projection share the same weight tensor. The shared tensor is exposed once through `parameters()`, so optimisers and checkpoints store it once.

### Token
An integer ID representing a byte, subword, or special symbol in the BPE vocabulary.

### Tokenizer
The module (`src/tokenizer.rs`) providing byte-level BPE training, encoding, decoding, batch encoding, and JSON persistence. Training accepts a single byte slice, multiple borrowed byte shards, memory-mapped shard files, or a directory of shard files; shard and configured pre-tokenization boundaries are never crossed by merges.

### TokenizerQualityReport
Held-out tokenizer metrics from `src/tokenizer_quality.rs`: compression ratio (bytes/token), tokens per character, unknown-token rate, vocabulary utilization, and encode/decode throughput. Exposed via `--tokenizer-report` and emitted during training on the validation partition.

### Top-k Sampling
A sampling strategy that restricts the candidate token set to the `k` tokens with the highest logits before applying softmax and sampling. Configured in `SamplingConfig::top_k`.

### Top-p Sampling
See **Nucleus Sampling**.

### Topological Order
The order of tensors from a given root to all its ancestor leaves such that every parent appears after its children. Computed by `Tensor::topological_order` using a post-order DFS. The backward pass iterates this order in reverse.

### TrainConfigFile
JSON training configuration (`src/config.rs`) with sections `model`, `tokenizer`, `trainer`, `optimizer`, `schedule`, and `data`. Loaded via `--config`; schema at `training/schema.json` (regenerate with `oxidelm --train init --schema`). `oxidelm --train init` writes a starter file from an [`ArchitecturePreset`](#architecturepreset) plus `--memory-budget`, applying the same auto-fit knobs as `--fit-memory`.

### TrainingBackend
An enum (`Cpu` | `Gpu(GpuContext)`) selecting where training steps are executed.

### TrainingMetrics
A struct emitted per optimizer step, containing: step, loss, perplexity, learning rate, gradient norm, tokens per second, loss scale, and overflow flags. Optional JSONL export (`--metrics-jsonl` / `trainer.metrics_jsonl`) writes `step`, `loss`, `ppl`, `lr`, `grad_norm`, `loss_scale`, and `rss` for external plotting.

### Transfer (Op::Transfer)
A graph node recording a host↔device transfer. The backward pass simply propagates gradients through it unchanged.

### TransformerBlock
A pre-normalised decoder block (`src/transformer.rs`) wrapping: `RMSNorm → MultiHeadAttention → residual add → RMSNorm → FeedForward (SwiGLU) → residual add`. Implements `forward`, `forward_gpu`, `forward_cached`, and `forward_cached_gpu`.

### TTFT (Time To First Token)
The wall-clock time from prompt submission through model forward to the first sampled token. Used as a key latency metric for interactive applications.

---

## U

### UNK (Unknown Token)
A special token (ID 257) used to represent unseen or out-of-vocabulary bytes during decoding. Stored in `SpecialTokens`.

---

## V

### V (Vocabulary Size)
The number of tokens in the BPE vocabulary. The LM head output dimension equals `V`, producing logits over all possible tokens.

### Value (V)
In self-attention, each input token is projected to a value vector whose weighted sum produces the attention output. Projected via `v_proj`.

---

## W

### wgpu
A cross-platform Rust graphics and compute API, serving as OxideLM's GPU backend. Targets **Metal** (Apple), **Vulkan** (Linux/Windows), **DX12** (Windows), and **WebGPU** (browsers). Does **not** use CUDA directly.

### WGSL (WebGPU Shading Language)
The shader language used for all OxideLM GPU kernels. Shader source files live in `shaders/` and are statically embedded in the binary via `include_str!`.

### Weight Decay
An L2 regularisation term in AdamW applied directly to parameter values (not via the gradient): `value -= learning_rate * weight_decay * value`. Controlled per parameter group; defaults exclude RMSNorm scales and selected embeddings.

### Workgroup
A grid of GPU threads executing the same compute shader. OxideLM matmul kernels use `16×16` or `8×32` workgroup sizes; element-wise and reduction kernels use 256 threads per workgroup.

---

## X

### Xavier Initialization (Glorot Initialization)
Weight initialisation that samples uniformly from `[-√(6/(fan_in + fan_out)), √(6/(fan_in + fan_out))]`. Used for all linear and embedding weights in OxideLM.

---

## Z

### Zero-Copy
A technique where data is accessed in-place via pointers or memory maps rather than being copied into a new allocation. OxideLM tokenizer training memory-maps input files, and GPU execution reuses cached upload buffers until parameters are invalidated.

### Zero Gradients
The operation of clearing all accumulated gradients in the computation graph before a new forward/backward training iteration. `Tensor::zero_grad()` traverses the topological order and sets `grad` and `gpu_grad` to `None`.