# OxideLM Glossary
A reference of technical terms used throughout the OxideLM project. Each entry includes a name and a succinct explanation of how the term is used in this codebase.
---
## A
### Activation Function
A non-linear function applied element-wise to tensor values. OxideLM uses **ReLU**, **Sigmoid**, **GELU**, and **SiLU** (the latter in the SwiGLU feed-forward block).
### AdamW
An adaptive stochastic optimisation algorithm that combines **Adam** with decoupled **weight decay**. Maintains per-parameter first and second moment estimates (`beta1`, `beta2`) and applies bias correction and L2 regularisation independently of the adaptive learning rate. Implemented in `src/optimizer.rs` for both CPU and GPU.
### AdamWConfig
The hyperparameter record for the AdamW optimiser: `beta1`, `beta2`, `epsilon`, `weight_decay`, plus parameter-group controls (`no_decay_rms_norm`, `no_decay_embeddings`, `rms_norm_lr_scale`, `embedding_lr_scale`). Stored in checkpoints and serialised with Serde.
### AdamWParams
A `#[repr(C)]` struct (`gpu.rs`) holding all AdamW hyperparameters and the per-step corrections, uploaded as a uniform buffer to the GPU optimiser kernel.
### AdamWState
The persisted optimiser state: a step counter and a vector of `ParameterMoments` (first- and second-moment `Vec<f32>`). Saved in checkpoints and reconstructed when training resumes.
### ArchitecturePreset
Named model-size templates (`llama-tiny`, `llama-micro`, `llama-small`, `gpt-tiny`, `gpt-micro`, `gpt-small`) in `src/model.rs`. Each expands to hidden size, layers, heads, KV heads, intermediate size, context length, bias, and tied-embedding defaults. `*-micro` targets ≤8 GiB FP32 training with activation checkpointing. Used by `--architecture` / `model.architecture` and by `oxidelm --train init`.
### AttentionKvCache
A struct holding the **key** and **value** tensors accumulated from past tokens during autoregressive generation. Each transformer layer has its own optional cache, enabling incremental decoding without recomputing the full history. `truncate` drops rejected speculative draft positions.
### Autograd (Automatic Differentiation)
The reverse-mode automatic differentiation engine in `src/lib.rs`. Every operation records an `Op` variant on its output tensor. Calling `.backward()` on a scalar loss tensor performs a post-order topological traversal over the computation graph, computing gradients through each operation in reverse.
### Autoregressive Decoding
A generation mode where the model produces one token at a time, feeding the newly sampled token back as input for the next step. Implemented via `forward_cached` and `forward_cached_gpu` to reuse the **KV cache**.
---
## B
### B (Batch Size)
The first dimension of input tensors (`[B, S, ...]`). Number of independent sequences processed simultaneously. In training config this is the **micro-batch** width (`trainer.batch_size` / `--batch-size`), not the effective batch seen by the optimizer — see [Effective Batch Size](#effective-batch-size).
### Backward Pass
The process of computing gradients by traversing the computation graph in reverse topological order, starting from a scalar loss and propagating gradients to leaves. Entry point: `Tensor::backward()`.
### BackwardParams
A `#[repr(C)]` uniform-buffer struct carrying shape metadata and scalar values to the GPU backward kernels.
### Batched GEMM
General Matrix Multiply applied to batched tensors. OxideLM supports `[B, H, M, K] × [B, H, K, N] → [B, H, M, N]` as `B * H` independent products in one dispatch. The fused attention path instead combines score calculation, online Softmax, and value aggregation without storing the score matrix.
### BOS (Beginning Of Sequence)
A special token (ID 258) optionally prepended to encoded sequences. Stored in `SpecialTokens`.
### Broadcasting
The NumPy-style trailing-dimension broadcasting for element-wise binary operations. When two tensor shapes differ, dimensions of size 1 are stretched to match the other operand. Implemented in `broadcast_shape` and `broadcast_offset` in `lib.rs`.
### BPE (Byte-Pair Encoding)
A subword tokenisation algorithm that starts from raw bytes and iteratively merges the most frequent adjacent token pair. OxideLM implements **byte-level** BPE, meaning the base vocabulary is the 256 single-byte values.
### BpeTokenizer
The main tokenizer struct (`src/tokenizer.rs`) holding the vocabulary (`vocab`), encoder mapping (`encoder`), merge rules (`merges`), merge priority ranks (`merge_ranks`), and special token IDs. Train via `train_from_bytes*`, `train_from_file*`, `train_from_byte_shards*`, `train_from_shards*`, or `train_from_shard_directory*`.
### BYTE_VOCAB_SIZE
A constant equal to 256 — the number of base single-byte tokens in the **byte-level BPE** vocabulary.
---
## C
### Causal Attention
Attention that prevents a token from attending to future positions. Enforced by adding a causal mask (upper-triangular `-∞` entries) before softmax.
### CausalLM (Causal Language Model)
The top-level model struct (`src/model.rs`): a decoder-only Transformer comprising a token embedding, a stack of `TransformerBlock` layers, a final `RmsNorm`, and an LM head projection. Exposes `forward`, `forward_gpu`, `forward_cached`, training helpers, and generation methods.
### CausalLmConfig
The model hyperparameter record, including model widths, query and key/value head counts, context length, normalization, RoPE, initialization, and embedding tying options.
### Causal Mask
An additive mask applied to attention scores before softmax: positions where the query comes before the key get `-∞`, forcing attention weights to zero after exponentiation. For cached decoding the mask also accounts for already-consumed past tokens.
### Activation Checkpointing
A training memory optimisation that discards intermediate activations inside each `TransformerBlock` during the forward pass and recomputes them during backward. Enabled via `TrainerConfig::activation_checkpointing` / `--activation-checkpointing`. Implemented with `checkpoint` / `checkpoint_gpu` and `Op::Checkpoint`.
### Checkpoint
A serialised snapshot of named model weights and optimizer moments (binary LE `F32` payloads with per-tensor CRC32 checksums), tokenizer state, learning-rate schedule, RNG state, data-loader state, trainer position, best validation loss, and consecutive non-improving validation evaluations. Written atomically (`*.tmp` → validate → rename) in the `OXLMCKP2` binary format; loaded with `load_checkpoint`. Legacy JSON float-array checkpoints (version 1) still load.
### Continuous Batching
Serving scheduler (`ContinuousBatcher` in `src/generate.rs`) that batches active decode requests whose KV positions match, lets finished sequences leave the batch, and admits waiting prompts into free slots. `Generator::generate_batch` is the convenience entry point.
### Corpus
Training text discovery (`src/corpus.rs`): a single `.txt` file, a JSONL file of `{"text":…}` records, or a directory of `.txt` / `.jsonl` files. Multi-file and JSONL inputs are concatenated in sorted path order into the token-cache `source/` directory so validation split and token sharding are deterministic.
### CpuMatmulBackend
An enum (`Reference` or `BlockedParallel`) controlling which CPU matrix multiplication implementation is used globally or per call. The default is `BlockedParallel`.
### Cross-Entropy Loss
The standard categorical loss: for each row of logits the negative log-likelihood of the correct class is computed and averaged across all rows. Implemented in `Tensor::cross_entropy` (CPU) and as a GPU kernel in `index.wgsl`.
---
## D
### D (Hidden Size / Model Dimension)
The feature dimension of the Transformer (`hidden_size`). Every token is represented by a vector of length `D` throughout the model.
### Decoder-Only Transformer
A Transformer architecture without an encoder stack. Input tokens attend only to themselves and previous positions (causal attention). Used by GPT-style language models. OxideLM implements this pattern.
### Data Parallel (CPU)
Training with multiple worker threads, each owning a private model replica (`src/data_parallel.rs`). The micro-batch is split along the batch axis, workers run forward/backward independently, and gradients are mean-reduced onto the primary model before the optimizer step. Graphs stay single-threaded (`!Send`); only weight/gradient `Vec<f32>` buffers cross threads. CPU uses `CpuDataParallelPool`; GPU uses `GpuDataParallelPool` with one `GpuContext` per worker from `GpuContext::create_devices` (distinct adapters when available, else multiple logical devices). Configured by `TrainerConfig::data_parallel_workers` (batch size must be ≥ workers and divisible by the worker count). Does **not** multiply [effective batch size](#effective-batch-size); it only parallelizes each micro-batch of size `batch_size`.
### Device
An enum (`Cpu` | `Gpu`) tracking where a tensor's authoritative data currently resides.
### Dispatch
A single GPU kernel invocation. OxideLM launches compute dispatches via `wgpu::ComputePass::dispatch_workgroups`.
---
## E
### ElementParams
A `#[repr(C)]` uniform-buffer struct carrying shape parameters (`length`, `rows`, `columns`, `epsilon`) to element-wise and normalisation GPU kernels.
### Element-Wise Operation
An operation applied independently to each element of one or two tensors (add, sub, mul, div, pow, relu, sigmoid, gelu, silu). GPU element-wise kernels use 256 threads per workgroup.
### Embedding
A lookup table that maps integer token IDs to dense vectors of size `hidden_size`. Implemented in `src/layers.rs`; the `Embedding::forward` method delegates to `Tensor::embedding`.
### EOS (End Of Sequence)
A special token (ID 259) that marks the end of a sequence. During generation, sampling stops when this token is produced.
### Early Stopping
Training halt triggered after a configurable number of consecutive validation evaluations fail to improve best validation loss by at least `early_stopping_min_delta`. Controlled by `early_stopping_patience` (0 disables). Decisions never use training loss.
### Effective Batch Size
Sequences contributing to one optimizer step: `batch_size × gradient_accumulation_steps`. Logged as `effective_batch_size` in the train / dry-run plan. `data_parallel_workers` shards each micro-batch and does not appear in the product. Peak activation memory tracks micro-batch size (per-replica `batch_size / data_parallel_workers` when workers > 1); raise `gradient_accumulation_steps` to grow the statistical batch under a RAM budget. See also [Gradient Accumulation Steps](#gradient-accumulation-steps) and [Data Parallel (CPU)](#data-parallel-cpu).
### EvaluationMetrics
A struct carrying validation results: step, loss, perplexity, token counts, wall time, whether the model improved, path to the best checkpoint, consecutive non-improving evaluations, and whether early stopping fired.
---
## F
### FeedForward
The SwiGLU feed-forward sub-layer inside a Transformer block (`src/transformer.rs`). Contains `gate_proj`, `up_proj`, and `down_proj` linear projections.
### Frequency Penalty
An additive sampling penalty: each token's logit is reduced by `frequency_penalty * count`, where `count` is how often that token appears in the full history.
### FIRST_MERGE_ID
The first token ID assigned to a learned BPE merge (value 260). IDs 0–255 are raw bytes, 256–259 are special tokens.
### BF16 / FP16 / FP32
Floating-point storage formats. OxideLM's normal tensor path uses FP32. Its mixed-precision matmul path supports native FP16 storage when the adapter exposes `SHADER_F16`, portable packed BF16 storage, and an FP32 fallback. All mixed-precision dot products accumulate and output in FP32.
### Mixed-precision training (AMP)
Optional training mode (`MixedPrecisionConfig` / `--mixed-precision`) that keeps FP32 master weights and AdamW state while rounding working weights to FP16 or BF16. Loss, Softmax, RMSNorm, and gradient-norm reductions stay FP32. A dynamic loss scaler multiplies the loss before backward, unscales gradients before the optimizer, skips the step on overflow, and adjusts the scale.
### FusionParams
A `#[repr(C)]` uniform-buffer struct carrying metadata to fused GPU kernels (e.g., residual + RMSNorm, SwiGLU, masked softmax).
---
## G
### GELU (Gaussian Error Linear Unit)
An activation function: `0.5 * x * (1 + tanh(√(2/π) * (x + 0.044715 * x³)))`. Used as an alternative to ReLU in some Transformer variants.
### Generate
The module (`src/generate.rs`) providing text generation with configurable sampling: temperature, top-k, top-p, repetition / frequency / presence penalties, seeded randomness, continuous batching via `ContinuousBatcher`, and speculative decoding via `SpeculativeGenerator`.
### Gradient
The partial derivative of the scalar loss with respect to a specific tensor element. Leaf tensors store gradients in the `grad` field (CPU `Vec<f32>`) or `gpu_grad` field (GPU buffer).
### Gradient Accumulation
When a tensor is used multiple times in a computation graph, its gradient contributions are summed (accumulated) rather than overwritten. Handled by `accumulate_gradient` in `lib.rs`. Distinct from the trainer knob [Gradient Accumulation Steps](#gradient-accumulation-steps).
### Gradient Accumulation Steps
Trainer knob (`trainer.gradient_accumulation_steps` / `--gradient-accumulation-steps`): number of micro-batches per optimizer step. Each micro-batch runs forward/backward with gradients scaled by `1/steps`; after all steps, AdamW runs once. Multiplies [effective batch size](#effective-batch-size) without increasing micro-batch activation peak. Orthogonal to `data_parallel_workers`, which only shards each micro-batch.
### Gradient Clipping
Scaling down gradients when the global norm exceeds `max_gradient_norm`, preventing exploding gradients. Applied in the training loop.
### Gradient Norm
The L2 norm (Euclidean length) of all parameter gradients concatenated. Reported in training logs and used for gradient clipping.
---
## H
### H (Number of Attention Heads)
The dimension representing the number of independent attention heads. The hidden size `D` must be divisible by `H`, yielding a per-head dimension of `D/H`.
### head_dim
The feature dimension of a single attention head: `hidden_size / num_heads`. Must be even because RoPE operates on adjacent pairs in this dimension.
---
## I
### IndexTensor
An integer-only tensor (`Vec<usize> + shape`) for token IDs and class indices. Unlike `Tensor`, it stores exact integers rather than `f32`, enabling direct indexing into embeddings and cross-entropy targets. GPU execution uses typed `u32` index buffers.
### Initialization
The module (`src/initialization.rs`) providing normal, **Xavier uniform**, and **Kaiming uniform** weight initialisation. Full language models use configurable normal initialization; standalone linear and embedding layers retain Xavier initialization.
### Initializer
A seeded `StdRng` wrapper that produces either normal or Xavier-initialised tensors. Residual-output tensors can apply an extra depth-dependent scale.
---
## K
### Kaiming Initialization (He Initialization)
Weight initialisation that samples from `[-√(6/fan_in), √(6/fan_in)]`. Provided in `initialization.rs` for ReLU-style activations.
### Key (K)
In self-attention, each input token is projected to a key vector that determines which positions attend to it. Projected via `k_proj`.
### KV Cache
Short for **Key-Value Cache**. See **AttentionKvCache**.
---
## L
### Layers
The module (`src/layers.rs`) providing `Linear` and `Embedding` building blocks with both CPU and GPU forward implementations.
### Leaf
An `Op::Leaf` variant marking a tensor that was created directly (not as the result of an operation). Leaf tensors are the targets of gradient accumulation during `backward()`.
### LearningRateSchedule
A cosine decay schedule with linear warmup. During warmup steps the learning rate increases linearly to `base_learning_rate`; afterwards it follows a cosine curve down to `minimum_learning_rate`.
### Linear
A linear (fully-connected) layer: `output = input @ weight`. Supports standard weight storage (`[in_features, out_features]`), transposed weight sharing (used for weight tying), and inference-only int8/int4 weight copies with per-output-channel scales. Implemented for CPU (`forward`) and GPU (`forward_gpu`).
### LM Head
The final linear projection (`lm_head`) that maps the final hidden state to vocabulary-sized logits. When `tie_word_embeddings` is enabled, it shares its weight tensor with the input embedding layer.
### Logits
The raw, unnormalised scores output by the LM head, of shape `[B, S, V]`. Fed into cross-entropy loss during training or softmax for sampling during generation.
---
## M
### Memory Budget
Static FP32 footprint planning in `src/memory.rs`. Reports parameter, activation, KV-cache, staging/readback, and optimizer bytes for a model and batch shape, then refuses configurations that exceed the selected CPU/GPU budget or `max_buffer_size`.
### Matmul (Matrix Multiplication)
The product of two 2D matrices (or batched stacks thereof). CPU: `Tensor::matmul` with a choice of `Reference` (triple loop) or `BlockedParallel` (tiled, Rayon-parallelised) backend. GPU: dispatched via WGSL tiled kernels (`matmul.wgsl`, `matmul_vec4.wgsl`).
### MatmulDimensions
A struct computed during matrix multiplication holding the tensor rank, batch count, rows, shared/inner dimension, and columns.
### MatmulParams
A `#[repr(C)]` uniform-buffer struct carrying `(M, K, N, batch)` to the GPU matmul kernel.
### mmap / memmap2
Memory-mapped file I/O. The tokenizer uses `memmap2` to read training text directly from disk without copying it into a Rust `Vec<u8>`, keeping memory usage proportional to the working token-ID array rather than the corpus size. Multi-file shard training maps each file separately and never concatenates shard bytes.
### Moments (Optimizer)
First moment (moving average of gradients) and second moment (moving average of squared gradients) maintained per parameter by **AdamW**. Stored in `ParameterMoments`.
### MultiHeadAttention
The grouped-query self-attention module (`src/attention.rs`). Projects input into Q/K/V, maps groups of query heads to shared K/V heads, applies RoPE and causal attention, then merges query heads and projects back via `o_proj`.
---
## N
### Normalization
See **RMSNorm**.
### Nucleus Sampling (Top-p Sampling)
A sampling strategy that restricts tokens to the smallest set whose cumulative probability mass exceeds `top_p` (e.g., 0.95), preventing long-tail low-probability tokens from being sampled. Configured in `SamplingConfig`.
---
## O
### Op (Operation)
An enum variant in `src/lib.rs` recording the computation that produced a tensor (e.g., `Add`, `MatMul`, `SoftmaxLast`, `Rope`, `CausalMask`, `Embedding`). Used during `backward()` to dispatch the correct gradient computation.
### Optimizer
The module (`src/optimizer.rs`) providing the `AdamW` struct with `step` (CPU) and `step_gpu` methods, state management, and checkpoint persistence.
---
## P
### Parameter Group
Per-parameter AdamW settings (`weight_decay`, `lr_scale`) derived from tensor names. By default RMSNorm scales and selected embeddings (`tok_embeddings.weight`, `lm_head.weight`) receive zero weight decay; group learning-rate scales multiply the schedule LR. Built via `AdamW::new_grouped` / `from_state_grouped`.
### PAD (Padding Token)
A special token (ID 256) used to fill sequence positions that should be ignored by attention. Stored in `SpecialTokens`.
### ParameterMoments
A struct holding the first- and second-moment `Vec<f32>` for a single parameter in the AdamW optimiser.
### Permute
A tensor operation that reorders dimensions. For example, `permute([0, 2, 1, 3])` on `[B, S, H, D/H]` produces `[B, H, S, D/H]`, used to split and merge attention heads.
### Perplexity
The exponential of the cross-entropy loss: `exp(loss)`. A standard intrinsic evaluation metric for language models; lower perplexity means the model assigns higher probability to the correct tokens.
### Prefill
The initial forward pass that processes the entire input prompt before generating tokens. During prefill the KV cache is populated and attention is computed over the full prefix.
### Presence Penalty
An additive sampling penalty: each token that appears at least once in the history has `presence_penalty` subtracted from its logit.
### Prompt
The input text that conditions generation. Encoded into token IDs and fed as the initial input to `CausalLM::generate`.
---
## Q
### Quality evaluation
Standalone checkpoint metrics from `src/quality.rs` / `--eval`: held-out cross-entropy loss and perplexity on fixed non-overlapping windows, top-1 next-token accuracy on the same windows, and a small greedy-completion task suite (built-in Shakespeare prompts or a JSON `{id, prompt, expected}` list) scored by exact match, prefix match, and leading-byte overlap.
### Quantized Linear Weight
An inference copy of a Linear weight: packed codes plus FP32 scales (and optional zero-points). Supports int8/int4/int3/int2, NF4, and FP8 with per-tensor / per-channel / per-group granularity. Round-trips via OxideLM checkpoints, `--export-quantized` SafeTensors (`{module}.weight.qpacked|.qscales|.qzeros`), and GGUF import/export.
### Quantization schema version
`CheckpointQuantization.schema_version` in `OXLMCKP2` metadata (`QUANTIZATION_SCHEMA_VERSION`, currently **3**). Loaders accept `1..=3` and reject `0` or newer versions with `unsupported quantization schema version N (supported 1..=V)`. Unknown `method` / format / granularity strings are also rejected. v3 adds optional `activation` and `kv_cache` blocks for W8A8/W4A8 and quantized KV.
### Quantization scheme matrix
Shipped weight packs: **int8/int4/int3/int2** (sym/asym), **NF4**, **FP8 E4M3/E5M2**; granularity **tensor / channel / group:{32,64,128}**; scales **absmax|minmax|percentile|mse**. Product methods: RTN, GPTQ, AWQ, SmoothQuant, HQQ, SpQR, QuIP#, AQLM. Inference modes: W8A8, W4A8, W4A16. See README “Scheme matrix” and `perf/baselines/quantize_schemes.json`.
### Round-to-Nearest (RTN)
Default `--method rtn` packer: per-block absmax (or other scale method) + nearest integer (or NF4/FP8 codebook). No calibration. Correctness bounds live in `tests/quant_correctness.rs` (`max_abs_dequant_error` ≤ `dequant_error_bound`).
### Quantization-Aware Training (QAT)
Training path that fake-quantizes Linear weights in the forward pass while using a straight-through estimator so gradients update the FP32/BF16 masters. Configure via trainer `qat` (`"int8"` / `"int4"` or `{ "format", "group_size", "learnable_scales" }`) or `--qat` / `--qat-group-size` / `--qat-learnable-scales`. Learnable scales use an LSQ-style `q * s` path (per-channel). After training, `oxidelm --export-qat` (or `CausalLM::export_qat_packed`) packs masters into inference weights.
### QLoRA
Frozen NF4/int4 base weights plus LoRA/DoRA adapters in higher precision. Set `lora.base_quant` (`nf4` / `int4`) and optional `lora.base_group_size` (default 64), or `--qlora-base` / `--qlora-group-size`. Adapters save/load separately via `--export-adapters` / `--import-adapters`.
### QKV Projection
Quadratic projection: projecting the input simultaneously into query, key, and value spaces. In OxideLM this is done via three separate `Linear` layers (`q_proj`, `k_proj`, `v_proj`). A fused QKV WGSL kernel was benchmarked but rejected as slower than the separate approach.
### Query (Q)
In self-attention, each input token is projected to a query vector used to compute attention scores against all keys. Projected via `q_proj`.
### QuIP#
Incoherence-processed weight quantization (`--method quip`). Applies a Walsh–Hadamard transform, RTN-packs, then folds the inverse back into a native packed Linear.
### GGUF
llama.cpp weight container. `--export-gguf` / `--import-gguf` support `Q4_0`, `Q4_K`, `Q5_K`, `Q6_K`, and `Q8_0`. Native mapping: `Q8_0`→`int8/group:32/sym`, `Q4_0`/`Q4_K`→`int4/group:32/asym`; `Q5_K`/`Q6_K` import as FP32 (no native 5/6-bit pack).
### HQQ
Half-Quadratic Quantization (`--method hqq`). Calibration-light alternating scale/zero-point optimization; requires `--asymmetric`.
### SpQR
Sparse-quantized residual path (`--method spqr`): isolate magnitude outliers, quantize the dense body, re-pack. For high-compression when a PPL budget still holds.
### AQLM
Additive quantization (`--method aqlm`): residual multi-codebook approximation folded into a native packed scheme.
---
## R
### Rayon
A Rust library for data parallelism. OxideLM uses Rayon for parallel pair-frequency counting in BPE training and for distributing output rows in the blocked-parallel CPU matmul.
### ReLU (Rectified Linear Unit)
The activation function `max(0, x)`. Available as `Tensor::relu()`.
### Repetition Penalty
A generation parameter that penalises tokens that have already appeared in the generated sequence by dividing their logits by `repetition_penalty` (typically ~1.1), reducing the likelihood of repetitive output. Optional `repetition_penalty_window` limits the history to the last N tokens.
### Reshape
A tensor operation that changes the shape without changing the number of elements or the row-major memory layout. Used extensively to flatten batches or reshape attention heads.
### Residual Connection
A path that adds the input of a sub-layer to its output (`x = x + sublayer(x)`). OxideLM Transformer blocks use two residual connections: one around attention and one around the feed-forward network.
### Reverse-Mode Differentiation
A form of automatic differentiation that computes the gradient of a scalar output with respect to all inputs in a single backward traversal. OxideLM implements this via topological sorting of the computation graph and per-operation gradient rules.
### RMSNorm (Root Mean Square Normalization)
A layer normalisation variant that normalises by the root mean square of activations (rather than variance), then applies a learnable per-feature scale. Used before attention and feed-forward sub-layers. Implemented in `src/norm.rs` with both CPU (`forward`) and GPU (`gpu_rms_norm`) paths.
### RoPE (Rotary Position Embeddings)
A positional encoding method that applies a rotation matrix to adjacent pairs of features in query and key vectors based on their absolute position. The rotation frequency (`theta`) controls the base wavelength. Implemented in `Tensor::rope` (CPU) and `gpu_rope_with_offset` (GPU).
### Row-Major
The memory layout used for all tensors: the last dimension varies fastest in the flat `Vec<f32>` buffer. `strides` are computed to support multi-dimensional indexing.
---
## S
### S (Sequence Length)
The second dimension of input tensors (`[B, S, ...]`), representing the number of tokens in each batch element.
### SamplingConfig
A struct holding generation decoding parameters: `max_new_tokens`, `temperature`, `top_k`, `top_p`, `repetition_penalty`, `repetition_penalty_window`, `frequency_penalty`, `presence_penalty`, `seed`, `return_logprobs`, and `top_logprobs`. Speculative decoding adds a separate `SpeculativeConfig` (`num_draft_tokens`).
### Scaled Dot-Product Attention
The core attention mechanism: `softmax(Q @ K^T / √d) @ V`, where `d` is the per-head dimension. OxideLM evaluates this in K/V tiles with an online Softmax, so it stores one log-sum-exp value per query instead of full score and probability matrices.
### Seed
A `u64` value used to initialise the random number generator for deterministic weight initialisation and text generation.
### SGD (Stochastic Gradient Descent)
The simplest optimisation rule: `value -= learning_rate * gradient`. Available as `Tensor::sgd_step()`.
### Sigmoid
The activation function `1 / (1 + exp(-x))`. Available as `Tensor::sigmoid()`.
### SiLU (Sigmoid Linear Unit / Swish)
The activation function `x * sigmoid(x)`. Used as the gating function inside SwiGLU. Available as `Tensor::silu()`.
### Softmax
The function `softmax(x_i) = exp(x_i - max(x)) / Σ_j exp(x_j - max(x))`. Implemented for the last dimension (`softmax_last_dim`) with numerical stability via max subtraction.
### SpecialTokens
A struct containing the token IDs for `pad`, `unk`, `bos`, and `eos`. Defaults to 256–259.
### Speculative Decoding
Draft-then-verify generation (`SpeculativeGenerator` in `src/generate.rs`). A smaller draft model proposes up to γ tokens (`SpeculativeConfig::num_draft_tokens`); the target verifies them in one multi-token cached forward, accepts a prefix (greedy match or Leviathan rejection sampling), truncates rejected KV positions, and samples a correction or bonus token from the target distribution. Requires matching vocabulary and EOS IDs.
### Staging Buffer
A CPU-accessible GPU buffer used for reading back results. OxideLM maintains a pool of staging buffers (`staging_pool`) to amortise allocation costs across readbacks.
### Storage Buffer
A GPU buffer holding tensor data. OxideLM uses `wgpu::BufferUsages::STORAGE` buffers with `TrackedStorageBuffer` wrappers that decrement global memory counters on drop.
### Strides
A vector computed alongside each tensor's shape that records the product of all later dimensions. Used for converting multi-dimensional indices to a flat buffer offset and for broadcasting logic.
### SwiGLU
A gated activation unit used in the feed-forward sub-layer: `(x @ gate_proj) · silu(x @ gate_proj) ⊙ (x @ up_proj)`, then projected back via `down_proj`. The GPU path uses a fused `gpu_swiglu` kernel that is 23% faster than separate dispatch.
---
## T
### Temperature
A generation parameter that controls output randomness. Logits are divided by `temperature` before softmax: `temperature → 0` yields greedy (deterministic) sampling; higher values increase diversity.
### Tensor
The core data structure (`src/lib.rs`): an `Rc<RefCell<TensorData>>` holding a `Vec<f32>` (CPU) or `GpuTensor` (GPU), shape, strides, optional gradient, device marker, and the operation that produced it. The autograd graph is intentionally single-threaded (`!Send` / `!Sync`); Rayon and GPU parallelism apply only to kernels and host helpers that use plain buffers, not shared graph nodes.
### Tie Word Embeddings
When `CausalLmConfig::tie_word_embeddings` is `true`, the token embedding matrix and the LM head projection share the same weight tensor. The shared tensor is exposed once through `parameters()`, so optimisers and checkpoints store it once.
### Token
An integer ID representing a byte, subword, or special symbol in the BPE vocabulary.
### Tokenizer
The module (`src/tokenizer.rs`) providing byte-level BPE training, encoding, decoding, batch encoding, and JSON persistence. Training accepts a single byte slice, multiple borrowed byte shards, memory-mapped shard files, or a directory of shard files; shard and configured pre-tokenization boundaries are never crossed by merges.
### TokenizerQualityReport
Held-out tokenizer metrics from `src/tokenizer_quality.rs`: compression ratio (bytes/token), tokens per character, unknown-token rate, vocabulary utilization, and encode/decode throughput. Exposed via `--tokenizer-report` and emitted during training on the validation partition.
### Top-k Sampling
A sampling strategy that restricts the candidate token set to the `k` tokens with the highest logits before applying softmax and sampling. Configured in `SamplingConfig::top_k`.
### Top-p Sampling
See **Nucleus Sampling**.
### Topological Order
The order of tensors from a given root to all its ancestor leaves such that every parent appears after its children. Computed by `Tensor::topological_order` using a post-order DFS. The backward pass iterates this order in reverse.
### TrainConfigFile
JSON training configuration (`src/config.rs`) with sections `model`, `tokenizer`, `trainer`, `optimizer`, `schedule`, and `data`. Loaded via `--config`; schema at `training/schema.json` (regenerate with `oxidelm --train init --schema`). `oxidelm --train init` writes a starter file from an [`ArchitecturePreset`](#architecturepreset) plus `--memory-budget`, applying the same auto-fit knobs as `--fit-memory`.
### TrainingBackend
An enum (`Cpu` | `Gpu(GpuContext)`) selecting where training steps are executed.
### TrainingMetrics
A struct emitted per optimizer step, containing: step, loss, perplexity, learning rate, gradient norm, tokens per second, loss scale, and overflow flags. Optional JSONL export (`--metrics-jsonl` / `trainer.metrics_jsonl`) writes `step`, `loss`, `ppl`, `lr`, `grad_norm`, `loss_scale`, and `rss` for external plotting.
### Transfer (Op::Transfer)
A graph node recording a host↔device transfer. The backward pass simply propagates gradients through it unchanged.
### TransformerBlock
A pre-normalised decoder block (`src/transformer.rs`) wrapping: `RMSNorm → MultiHeadAttention → residual add → RMSNorm → FeedForward (SwiGLU) → residual add`. Implements `forward`, `forward_gpu`, `forward_cached`, and `forward_cached_gpu`.
### TTFT (Time To First Token)
The wall-clock time from prompt submission through model forward to the first sampled token. Used as a key latency metric for interactive applications.
---
## U
### UNK (Unknown Token)
A special token (ID 257) used to represent unseen or out-of-vocabulary bytes during decoding. Stored in `SpecialTokens`.
---
## V
### V (Vocabulary Size)
The number of tokens in the BPE vocabulary. The LM head output dimension equals `V`, producing logits over all possible tokens.
### Value (V)
In self-attention, each input token is projected to a value vector whose weighted sum produces the attention output. Projected via `v_proj`.
---
## W
### wgpu
A cross-platform Rust graphics and compute API, serving as OxideLM's GPU backend. Targets **Metal** (Apple), **Vulkan** (Linux/Windows), **DX12** (Windows), and **WebGPU** (browsers). Does **not** use CUDA directly.
### WGSL (WebGPU Shading Language)
The shader language used for all OxideLM GPU kernels. Shader source files live in `shaders/` and are statically embedded in the binary via `include_str!`.
### Weight Decay
An L2 regularisation term in AdamW applied directly to parameter values (not via the gradient): `value -= learning_rate * weight_decay * value`. Controlled per parameter group; defaults exclude RMSNorm scales and selected embeddings.
### Workgroup
A grid of GPU threads executing the same compute shader. OxideLM matmul kernels use `16×16` or `8×32` workgroup sizes; element-wise and reduction kernels use 256 threads per workgroup.
---
## X
### Xavier Initialization (Glorot Initialization)
Weight initialisation that samples uniformly from `[-√(6/(fan_in + fan_out)), √(6/(fan_in + fan_out))]`. Used for all linear and embedding weights in OxideLM.
---
## Z
### Zero-Copy
A technique where data is accessed in-place via pointers or memory maps rather than being copied into a new allocation. OxideLM tokenizer training memory-maps input files, and GPU execution reuses cached upload buffers until parameters are invalidated.
### Zero Gradients
The operation of clearing all accumulated gradients in the computation graph before a new forward/backward training iteration. `Tensor::zero_grad()` traverses the topological order and sets `grad` and `gpu_grad` to `None`.