oxidelm 0.2.0

Bare-metal pure-Rust Transformer engine with reverse-mode autograd, BPE tokenization, and wgpu acceleration
Documentation

OxideLM ๐Ÿฆ€

A bare-metal, pure-Rust Transformer engine featuring custom reverse-mode autograd, zero-copy BPE tokenization, and cross-platform wgpu acceleration.

Rust 2021 License ML framework dependencies Unsafe Rust

No Python. No PyTorch bindings. No hidden tensor runtime.

OxideLM is a from-scratch causal language-model stack written in safe Rust. It implements the path from raw UTF-8 text to generated tokens: byte-level BPE, row-major tensors, reverse-mode automatic differentiation, a decoder-only Transformer, GPU compute kernels, AdamW training, checkpointing, and sampling.

The project is designed for learning and experimentation at the systems layer. Every important mechanismโ€”strides, graph traversal, gradient accumulation, causal attention, normalization, optimization, and GPU dispatchโ€”is visible in the source.

[!NOTE] โ€œZero dependencyโ€ means zero Python and external ML-framework dependencies. OxideLM uses a small set of Rust infrastructure crates: wgpu, bytemuck, pollster, memmap2, rayon, serde, serde_json, safetensors, and rand.

Architecture

 Raw UTF-8 text
        โ”‚
        โ–ผ
 โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
 โ”‚ Byte-level BPE tokenizer     โ”‚  mmap input ยท parallel pair counts
 โ”‚ memmap2 + rayon              โ”‚  reversible byte vocabulary
 โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                โ”‚ token IDs [B, S]
                โ–ผ
 โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
 โ”‚ Autograd tensor engine       โ”‚  row-major Vec<f32> ยท dynamic strides
 โ”‚ reverse topological backward โ”‚  lazy, accumulated gradients
 โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                โ”‚ tensors
       โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
       โ”‚                  โ”‚
       โ–ผ                  โ–ผ
 โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
 โ”‚ CPU       โ”‚    โ”‚ WGSL GPU kernels  โ”‚  tiled matmul ยท activations
 โ”‚ safe Rust โ”‚    โ”‚ wgpu              โ”‚  softmax ยท RMSNorm
 โ””โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”˜    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
       โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                  โ–ผ
 โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
 โ”‚ Decoder-only CausalLM        โ”‚
 โ”‚ Embedding โ†’ N ร— Block โ†’ Norm โ”‚
 โ”‚ โ†’ language-model head        โ”‚
 โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
                โ”‚ logits [B, S, V]
        โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
        โ–ผ                โ–ผ
 Cross-entropy      Top-k / top-p
 โ†’ backward         โ†’ decoded text
 โ†’ AdamW

Each pre-normalized Transformer block follows:

x โ”€โ”€ RMSNorm โ”€โ”€ causal attention โ”€โ”€ + โ”€โ”€ RMSNorm โ”€โ”€ SwiGLU โ”€โ”€ + โ”€โ”€ output
โ”‚                                      โ–ฒ                        โ–ฒ
โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ residual โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜                        โ”‚
                         residual โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜

Attention projects [B, S, D] into [B, H, S, D/H], applies RoPE to queries and keys, masks future positions, performs scaled dot-product attention, and joins the heads back into [B, S, D].

๐Ÿ“– Glossary of technical terms

Why OxideLM?

Area What OxideLM provides
๐Ÿงฎ Explicit autograd Reverse-mode differentiation with topological graph traversal, on-demand gradient storage, and correct accumulation through branched graphs.
โšก Memory-conscious tokenization Training data is memory-mapped instead of copied wholesale. Pair-frequency work is distributed across CPU cores with Rayon.
๐Ÿง  Modern decoder design RMSNorm, rotary position embeddings, grouped-query causal attention, SwiGLU, configurable token-embedding/LM-head weight tying, and cross-entropy loss.
๐Ÿ”ข Integer token paths IndexTensor keeps token IDs exact as CPU integers and uploads typed u32 index buffers for GPU embedding and loss kernels.
๐Ÿ–ฅ๏ธ Parallel CPU matmul Safe blocked matrix multiplication distributes output rows with Rayon and retains the original triple loop as a selectable reference.
๐ŸŽฎ Portable GPU compute Persistent parameter buffers and device-resident WGSL matmul, fused residual + RMSNorm, SwiGLU, masked Softmax, and activation kernels.
๐Ÿ“ฆ Quantization Weight-only RTN/GPTQ/AWQ/SmoothQuant/HQQ/SpQR/QuIP#/AQLM (int8โ€“int2, NF4, FP8), mixed modes W8A8 / W4A8 / W4A16, KV-cache int8/FP8/NF4, plus GGUF + quantized SafeTensors interop.
๐Ÿ›ก๏ธ Safe systems code No project-authored unsafe; ownership and lifetimes remain explicit across tensor graphs and GPU resources.
๐Ÿ”ฌ Readable internals Compact modules expose the mechanics usually hidden behind Python APIs and mature ML frameworks.
๐Ÿ‹๏ธ End-to-end training Depth-scaled Transformer initialization, gradient clipping, cosine warmup scheduling, AdamW, resumable checkpoints, and throughput reporting.
โœ๏ธ Practical decoding Per-layer KV caching, one-token decode steps, temperature, top-k, nucleus sampling, repetition / frequency / presence penalties, seeded randomness, EOS stopping, continuous batching, and speculative decoding with a smaller draft model.

CPU matrix multiplication

Tensor::matmul uses CpuMatmulBackend::BlockedParallel by default. It blocks the shared and output-column dimensions, distributes independent output rows with Rayon, and falls back to the reference loop when a matrix has too few rows to benefit from threads. Select either implementation globally or per call:

use oxidelm::{set_cpu_matmul_backend, CpuMatmulBackend};

set_cpu_matmul_backend(CpuMatmulBackend::Reference);
let reference = left.matmul(&right);
let optimized = left.matmul_with_backend(&right, CpuMatmulBackend::BlockedParallel);

GPU execution

Host Vec<f32>
    โ†’ to_gpu() returns GPU Tensor
    โ†’ GPU storage buffer shared by reshape views
    โ†’ device-resident kernel chain
    โ†’ to_cpu() requests readback
    โ†’ pooled readback staging buffer

Matrix multiplication ships a safe 16 ร— 16 tiled kernel, an alignment-specialized 16 ร— 16 kernel without boundary branches, a benchmark candidate using 32 ร— 32 register blocking, and a vec4-loaded 8 ร— 32 kernel. On the first use of a matrix shape, automatic selection times every valid tile and workgroup configuration. The fastest median is cached by adapter identity, driver, backend, and matrix dimensions. Timestamp-query kernel time is used when available; synchronized wall time is the fallback. Element-wise and row-reduction kernels use 256 threads per workgroup. Mixed-precision matrix operands can use FP16 or packed BF16 storage while multiplying and accumulating in FP32. FP16 is enabled only when the adapter exposes SHADER_F16; otherwise the upload selects the FP32 kernel. Packed BF16 uses portable core WGSL bit operations. wgpu targets Metal, Vulkan, DX12, and browser WebGPU; NVIDIA hardware is reached through Vulkan or DX12 rather than a CUDA-specific backend.

Model parameters receive persistent device buffers through prepare_gpu, which forward_gpu calls automatically. Parameter updates invalidate those buffers, so the next GPU forward uploads the new values once. GPU operations do not read results back automatically. Tensor::data(), item(), and CPU math accept only CPU-resident tensors; call Tensor::to_cpu() at the wanted readback boundary. Likewise, Tensor::to_gpu() returns a GPU-resident tensor and reuses cached uploads. Multi-head attentionโ€”including head permutation, RoPE, tiled K/V reads, online causal Softmax, value aggregation, and KV-cache growthโ€”stays on the GPU without materializing a quadratic score matrix.

Linear weights can be quantized for inference. Weight-only paths keep activations in FP32; mixed modes (W8A8, W4A8) also quantize activations to int8 (static or dynamic, per-tensor or per-token). Optional KV-cache quantization packs keys/values as int8, FP8, or NF4 with per-head or per-layer scales:

use oxidelm::layers::WeightQuantization;

model.quantize_linear_weights(WeightQuantization::Int4);

Int8 packs four weights per u32; Int4 packs eight. Each logical output channel has its own symmetric scale. GPU preparation uploads the packed weights and scales instead of the corresponding FP32 Linear buffers. The original FP32 parameters remain on the CPU for checkpoints and can be restored as the active inference path with clear_quantized_linear_weights(). For the product CLI (--quantize, inference modes, calibration, KV packs), see Quantize a checkpoint.

For quantization-aware training (QAT), enable fake-quant + STE while keeping FP32/BF16 masters trainable (compatible with --mixed-precision):

cargo run --release --bin oxidelm -- \
  --train training/input.txt training/model.bin \
  --config training/tiny.json \
  --qat int8 --qat-group-size 64

Learnable scales (LSQ): --qat-learnable-scales or "learnable_scales": true in the trainer qat object. After training, pack with oxidelm --export-qat model.bin out.bin.

QLoRA freezes an NF4/int4 base and trains LoRA/DoRA adapters:

# in training JSON under trainer.lora:
# { "rank": 8, "alpha": 16, "base_quant": "nf4", "base_group_size": 64, "targets": ["q_proj","v_proj"] }
oxidelm --export-adapters model.bin adapters.safetensors
oxidelm --import-adapters model.bin adapters.safetensors model.adapted.bin

Token IDs and cross-entropy targets use IndexTensor, not floating-point Tensor. CPU embedding and loss code index integers directly. GPU execution checks the WebGPU u32 limit during the first upload, caches that typed buffer, and has no float-to-index conversion inside WGSL kernels.

Set CausalLmConfig::tie_word_embeddings to share the [vocab, hidden] token embedding with the transposed LM projection. Tied mode exposes the shared tensor once through parameters(), so SGD, AdamW, GPU AdamW, and checkpoints update or store it once. Older checkpoints without this field load in untied mode.

CausalLmConfig::initializer_std controls the normal distribution used for model weights and defaults to 0.02. Attention output and FFN down projections use initializer_std / sqrt(2 * num_layers) to keep residual activations and gradients stable as depth grows. Training JSON accepts model.initializer_std; the CLI override is --initializer-std.

Set CausalLmConfig::num_key_value_heads below num_heads to share each K/V head across a group of query heads. The query-head count must be divisible by the K/V-head count. K/V projections and caches use only num_key_value_heads * head_dim; training JSON and CLI use model.num_key_value_heads and --num-key-value-heads.

Attention score and value products use explicit batched GEMM: [B, H, M, K] ร— [B, H, K, N] -> [B, H, M, N]. The B * H matrices share one tiled GPU dispatch and never enter the CPU matmul path.

GPU training uses device-side cross-entropy and backward_gpu() kernels for matmul, RMSNorm, Softmax, SwiGLU, embeddings, RoPE, masking, permutation, and residual accumulation. AdamW parameters, gradients, and first/second moments remain in GPU buffers across steps. Only scalar loss and gradient-norm metrics are read back; checkpoint saving explicitly snapshots weights and moments to CPU storage.

Repeated dispatches reuse output storage from a shape-keyed LRU pool and reuse bind groups when the pipeline, buffers, and uniform parameters match. Defaults bound pooled outputs to 64 MiB and bind groups to 512 entries. A CLI --memory-budget of 8 GiB or less selects smaller defaults (16 MiB / 128 entries) via GpuContext::apply_memory_budget โ†’ set_cache_limits. Use set_cache_limits to change the bounds directly, trim_dispatch_caches to release them, and memory_stats to inspect allocations, reuse, cache hits, and retained bytes.

Check buffer residency and transfer counters on a machine with a GPU:

cargo run --release --example gpu_residency

Project layout

OxideLM/
โ”œโ”€โ”€ Cargo.toml
โ”œโ”€โ”€ Cargo.lock
โ”œโ”€โ”€ benches/
โ”‚   โ”œโ”€โ”€ cpu_matmul.rs            # reference vs blocked-parallel matmul
โ”‚   โ””โ”€โ”€ wgsl_bench.rs            # WGSL kernel harness and fusion comparisons
โ”œโ”€โ”€ examples/
โ”‚   โ”œโ”€โ”€ gpu_residency.rs         # buffer residency and transfer counters
โ”‚   โ”œโ”€โ”€ kv_cache_benchmark.rs    # prefill / decode benchmark with KV-cache
โ”‚   โ””โ”€โ”€ training_benchmark.rs    # fixed-shape training phase timings
โ”œโ”€โ”€ shaders/
โ”‚   โ”œโ”€โ”€ matmul.wgsl              # tiled matrix multiplication
โ”‚   โ”œโ”€โ”€ matmul_vec4.wgsl         # vec4-loaded 8ร—32 matmul variant
โ”‚   โ”œโ”€โ”€ matmul_fp16.wgsl         # FP16 product, FP32 accumulation
โ”‚   โ”œโ”€โ”€ matmul_bf16.wgsl         # packed BF16 input, FP32 accumulation
โ”‚   โ”œโ”€โ”€ matmul_quantized.wgsl    # packed int8/int4 Linear inference
โ”‚   โ”œโ”€โ”€ fusion.wgsl              # measured Transformer operation fusions
โ”‚   โ”œโ”€โ”€ index.wgsl               # integer embedding and loss kernels
โ”‚   โ”œโ”€โ”€ elementwise.wgsl         # add, activations, softmax, RMSNorm
โ”‚   โ”œโ”€โ”€ attention.wgsl           # RoPE, permutations, causal masking
โ”‚   โ”œโ”€โ”€ tiled_attention.wgsl     # online-Softmax tiled attention + backward
โ”‚   โ”œโ”€โ”€ backward.wgsl            # backpropagation kernels (matmul, RMSNorm, etc.)
โ”‚   โ””โ”€โ”€ optimizer.wgsl           # GPU-resident AdamW step
โ”œโ”€โ”€ src/
โ”‚   โ”œโ”€โ”€ lib.rs                   # tensor storage, operations, autograd
โ”‚   โ”œโ”€โ”€ tokenizer.rs             # mmap + parallel byte-level BPE
โ”‚   โ”œโ”€โ”€ tokenizer_quality.rs     # held-out compression / coverage / throughput reports
โ”‚   โ”œโ”€โ”€ quality.rs               # held-out perplexity / next-token accuracy / tiny task suite
โ”‚   โ”œโ”€โ”€ layers.rs                # linear and embedding layers
โ”‚   โ”œโ”€โ”€ norm.rs                  # RMSNorm
โ”‚   โ”œโ”€โ”€ attention.rs             # RoPE and causal multi-head attention
โ”‚   โ”œโ”€โ”€ transformer.rs           # SwiGLU and Transformer blocks
โ”‚   โ”œโ”€โ”€ model.rs                 # decoder-only CausalLM
โ”‚   โ”œโ”€โ”€ gpu.rs                   # wgpu runtime and graph integration
โ”‚   โ”œโ”€โ”€ initialization.rs        # normal, Xavier, and Kaiming initialization
โ”‚   โ”œโ”€โ”€ optimizer.rs             # AdamW
โ”‚   โ”œโ”€โ”€ data_loader.rs           # mmap token shards and batch prefetch
โ”‚   โ”œโ”€โ”€ data_parallel.rs         # CPU data-parallel replica pool + grad reduce
โ”‚   โ”œโ”€โ”€ corpus.rs                # .txt / JSONL / directory corpus discovery + concat
โ”‚   โ”œโ”€โ”€ trainer.rs               # batching, schedules, training loop
โ”‚   โ”œโ”€โ”€ checkpoint.rs            # binary named atomic checkpoints
โ”‚   โ”œโ”€โ”€ log.rs                   # quiet / human / JSON structured logging
โ”‚   โ”œโ”€โ”€ profile.rs               # named timing spans and aggregate profile reports
โ”‚   โ”œโ”€โ”€ memory.rs                # FP32 memory estimates and device-budget checks
โ”‚   โ”œโ”€โ”€ config.rs                # train JSON configs, CLI overrides, pre-alloc validation
โ”‚   โ”œโ”€โ”€ inspect.rs               # checkpoint parameter / memory / tokenizer reports
โ”‚   โ”œโ”€โ”€ analyze.rs               # versioned --analyze JSON report (ops / CI)
โ”‚   โ”œโ”€โ”€ weight_quant.rs          # RTN weight packs (int/NF4/FP8)
โ”‚   โ”œโ”€โ”€ act_quant.rs             # activation int8 + W8A8/W4A8/W4A16 modes
โ”‚   โ”œโ”€โ”€ kv_quant.rs              # KV-cache int8/FP8/NF4 packs
โ”‚   โ”œโ”€โ”€ calibration.rs           # calibration corpus / token-shard loader
โ”‚   โ”œโ”€โ”€ ptq.rs                   # GPTQ / AWQ / SmoothQuant
โ”‚   โ”œโ”€โ”€ advanced_quant.rs        # HQQ / SpQR / QuIP# / AQLM
โ”‚   โ”œโ”€โ”€ gguf.rs                  # GGUF Q4_0/Q4_K/Q5_K/Q6_K/Q8_0 I/O
โ”‚   โ”œโ”€โ”€ quantize.rs              # --quantize product surface + reports
โ”‚   โ”œโ”€โ”€ state_dict.rs            # named state dicts, SafeTensors, HF Llama + quantized packs
โ”‚   โ”œโ”€โ”€ generate.rs              # sampling, autoregressive / speculative decoding, continuous batching
โ”‚   โ””โ”€โ”€ main.rs                  # train / generate / inspect / analyze / quantize / eval / tokenizer / weight I/O CLI
โ”œโ”€โ”€ tests/
โ”‚   โ”œโ”€โ”€ tensor_engine.rs         # tensor math and analytical gradient checks
โ”‚   โ”œโ”€โ”€ tokenizer.rs             # BPE persistence and round-trip tests
โ”‚   โ”œโ”€โ”€ causal_lm.rs             # masking, shape, and gradient-flow tests
โ”‚   โ”œโ”€โ”€ gpu.rs                   # WGSL vs CPU numerical agreement checks
โ”‚   โ”œโ”€โ”€ gpu_verification.rs      # dedicated GPU kernel correctness suite
โ”‚   โ”œโ”€โ”€ full_model_correctness.rs # complete CPU/GPU logits, loss, and gradient comparison
โ”‚   โ”œโ”€โ”€ numerical_gradients.rs   # central finite-difference checks for every op
โ”‚   โ””โ”€โ”€ property_fuzz.rs         # proptest: shapes, tokenizer, checkpoints, sampling, GPU errors
โ””โ”€โ”€ training/
    โ”œโ”€โ”€ input.txt                # training corpus (sample text; also .jsonl or a directory)
    โ”œโ”€โ”€ tiny.json                # smoke / CI miniature config
    โ”œโ”€โ”€ low-ram-8gb.json         # 8 GiB RAM recipe (checkpointing + AMP bf16)
    โ”œโ”€โ”€ cpu-laptop.json          # conservative CPU-laptop recipe
    โ”œโ”€โ”€ gpu-16gb.json            # larger GPU recipe under ~16 GiB
    โ”œโ”€โ”€ schema.json              # JSON Schema for TrainConfigFile (`--train init --schema`)
    โ”œโ”€โ”€ model.bin                # saved model checkpoint
    โ””โ”€โ”€ model.best.bin           # best validation checkpoint

Quick start

Prerequisites

  • A current stable Rust toolchain with Cargo
  • A GPU and current graphics driver only when using --gpu
git clone https://github.com/MickyBalladelli/OxideLM.git
cd OxideLM
cargo test

Feature flags (lean builds)

OxideLM ships two opt-out Cargo features, both enabled by default:

Feature Enables Pulls in
gpu --gpu, forward_gpu/*_gpu methods, wgpu-backed tensors wgpu, pollster
training --train, Trainer, data loading, data-parallel, config resolution (no extra deps)

On constrained machines (no GPU, inference-only), build without either:

cargo build --release --no-default-features

This compiles a lean CPU tensor/tokenizer/--generate binary with no wgpu in the dependency tree. --train and --gpu exit with a clear error telling you which feature to rebuild with instead of failing to compile.

For CPU-only training (no GPU driver/wgpu required):

cargo build --release --no-default-features --features training

The default cargo build / cargo build --release keeps today's full behavior (gpu + training both enabled).

Train a miniature model

Generate a starter config from an architecture preset and memory budget (fits knobs automatically; default preset llama-tiny, default budget ~70% of host RAM). Presets: llama-tiny / gpt-tiny (smoke), llama-micro / gpt-micro (โ‰ค8 GiB FP32 + checkpointing), llama-small / gpt-small:

cargo run --release --bin oxidelm -- \
  --train init train.json \
  --architecture llama-micro \
  --memory-budget 6GiB

Write the JSON Schema for config files:

cargo run --release --bin oxidelm -- \
  --train init --schema training/schema.json

Smoke train from the repo root (paths are relative to the current working directory):

cargo run --release --bin oxidelm -- \
  --train training/input.txt training/model.bin \
  --config training/tiny.json

Train on ~8 GiB RAM:

cargo run --release --bin oxidelm -- \
  --train training/input.txt training/model.bin \
  --config training/low-ram-8gb.json \
  --memory-budget 6GiB

Train on a CPU laptop:

cargo run --release --bin oxidelm -- \
  --train training/input.txt training/model.bin \
  --config training/cpu-laptop.json \
  --fit-memory

Preview a memory plan without tokenizing or allocating weights:

cargo run --release --bin oxidelm -- \
  --train --config training/low-ram-8gb.json --dry-run

Override dimensions on the CLI (CLI flags win over the config file):

cargo run --release --bin oxidelm -- \
  --train input.txt model.bin \
  --config tiny.json \
  --hidden-size 128 \
  --num-layers 4 \
  --num-heads 8 \
  --intermediate-size 256 \
  --sequence-length 64 \
  --batch-size 2 \
  --data-parallel-workers 2 \
  --attention-dropout 0.1 \
  --residual-dropout 0.1 \
  --embedding-dropout 0.1 \
  --steps 200 \
  --lr 3e-4

Data-parallel training (--data-parallel-workers N, N > 1) splits each micro-batch across N replica threads, mean-reduces gradients, then steps the primary optimizer. Batch size must be divisible by N. On --gpu, workers use distinct adapters when available, otherwise multiple logical devices on one adapter (GpuContext::create_devices).

Effective batch size. One optimizer step consumes batch_size ร— gradient_accumulation_steps sequences (logged as effective_batch_size in the train / dry-run plan). The three knobs interact as follows:

Knob Role
batch_size (--batch-size) Micro-batch width: sequences per forward/backward. Peak activation memory scales with this.
gradient_accumulation_steps (--gradient-accumulation-steps) Micro-batches per optimizer step. Gradients are scaled by 1/steps and summed before AdamW. Multiplies effective batch without growing micro-batch activations.
data_parallel_workers (--data-parallel-workers) Shards each micro-batch across N replicas (batch_size must be โ‰ฅ N and divisible by N). Speeds the step; does not change effective batch size.

Example: batch_size=4, gradient_accumulation_steps=8, data_parallel_workers=2 โ†’ effective batch 32; each of two workers sees shard size 2 per micro-batch; eight micro-batches run before one optimizer step. Raise accumulation (not workers) when you want a larger statistical batch under a tight RAM budget. See also Effective Batch Size in the glossary.

Enable GPU-accelerated projections and normalization:

cargo run --release --bin oxidelm -- \
  --train input.txt model.bin \
  --config tiny.json \
  --gpu

Resume from a checkpoint (restores weights, AdamW moments, AMP loss scaler, activation-checkpoint interval, offload settings, schedule, sampler cursor, tokenizer, and model seed). --steps / --max-steps are aliases for the session step count (additional optimizer steps on resume). JSON schedule.total_steps is the LR schedule horizon and the default session length when the CLI omits them; if resume would pass that horizon, OxideLM extends it and prints a clear steps log line. When CLI and config disagree, the CLI wins for the session and a clarifying message is logged. Omit low-RAM CLI/JSON flags on resume, or keep them identical to the checkpoint โ€” mismatches error:

cargo run --release --bin oxidelm -- \
  --train input.txt model.bin \
  --resume \
  --steps 50

Run validation evaluation without further optimizer steps (--eval-only; often with --resume). Updates the best checkpoint when validation improves; leaves the final model.bin unchanged:

cargo run --release --bin oxidelm -- \
  --train input.txt model.bin \
  --resume \
  --eval-only

Training accepts a single .txt file, a JSONL file of {"text":โ€ฆ} records, or a directory of .txt / .jsonl files. Directories and multi-file / JSONL inputs are concatenated in sorted path order into the token cache (*.tokens/source/) so token sharding stays deterministic across runs. To continue on another corpus with the same checkpoint, pass the new input, keep --resume, and rebuild the token cache (the tokenizer stays the one stored in the checkpoint):

cargo run --release --bin oxidelm -- \
  --train part2.txt model.bin \
  --resume \
  --rebuild-token-cache \
  --steps 50

Examples:

# directory of text shards
cargo run --release --bin oxidelm -- --train corpus/ model.bin --config training/tiny.json

# JSONL documents
cargo run --release --bin oxidelm -- --train data.jsonl model.bin --config training/tiny.json

The trainer packs the corpus into non-overlapping sequences and shuffles them once per epoch. Shuffling is derived from the model seed, so runs and checkpoint resumes are reproducible. Attention, residual, and embedding dropout use step-derived deterministic masks during training and are disabled for evaluation and generation. Raw corpus bytes and encoded token shards are memory-mapped. Tokenization writes bounded chunks to model.bin.tokens/ by default, and a bounded background queue prepares upcoming batches. Override the cache with --token-cache, or tune data.shard_tokens, data.tokenize_chunk_bytes, and data.prefetch_batches in JSON (the matching CLI flags are --shard-tokens, --tokenize-chunk-bytes, and --prefetch-batches). The trainer then runs:

Long-context models can enable Hugging Face-style linear or YaRN RoPE scaling in the JSON model section. max_sequence_length must not exceed the original training context multiplied by factor:

{
  "model": {
    "max_sequence_length": 8192,
    "rope_scaling": {
      "rope_type": "yarn",
      "factor": 4.0,
      "original_max_position_embeddings": 2048
    }
  }
}

Use "linear" for rope_type to select linear position interpolation. YaRN also accepts optional attention_factor, beta_fast, and beta_slow values. Omitting rope_scaling keeps the original short-context RoPE behavior.

zero gradients โ†’ forward โ†’ cross-entropy โ†’ backward
               โ†’ clip gradients โ†’ AdamW step โ†’ checkpoint

The final part of the corpus is held out before tokenizer training. Evaluation uses fixed, non-overlapping validation sequences and never performs backward or optimizer steps. Logs report validation loss, perplexity, validation and training tokens processed, wall time, and the best checkpoint. By default the best model is saved beside model.bin as model.best.bin; override it with --best-checkpoint path.bin. Early stopping and best-checkpoint retention use validation loss only: set data.early_stopping_patience / --early-stopping-patience to stop after that many non-improving evaluations (0 disables), and optionally data.early_stopping_min_delta / --early-stopping-min-delta for the required decrease.

Training logs also report loss, perplexity, learning rate, gradient norm, tokens per second, step ETA, tokens seen, best validation loss (after the first eval), peak process RSS, and GPU live bytes when training on GPU. Choose output shape with --log human (default), --log json, or --log quiet. For external plotting, write one JSON object per optimizer step with --metrics-jsonl metrics.jsonl (or trainer.metrics_jsonl in the config): fields are step, loss, ppl, lr, grad_norm, loss_scale, and rss (peak RSS bytes, or null when unavailable). Library code emits structured events through oxidelm::log instead of hard-coded println! calls. Checkpoints are binary (OXLMCKP2): named tensors with shapes/dtypes/CRC32 checksums, optimizer moments, tokenizer state, schedule state, RNG and data-loader state, trainer position, and best validation loss. Saves write a temporary file, validate it, then rename.

Enable timing spans with --profile (summary) or --profile trace (each span plus summary). Equivalent env: OXIDELM_PROFILE=summary|trace. Named spans cover tokenizer encode/decode/train, data-batch loading, each transformer layer (layer.{i}), GPU upload/dispatch/synchronize, sampling, and checkpoint I/O. When profiling is off the guards are near zero cost; on exit the CLI flushes aggregated count / total / mean / max milliseconds per span.

Before allocating model weights, training resolves the config file and CLI flags, validates architecture / schedule / optimizer settings, prints an FP32 memory plan (parameters, activations, KV cache, staging, optimizer), and refuses configs that obviously exceed the selected device budget. When --memory-budget is omitted on CPU, the soft total defaults to about 70% of detected usable host RAM (falling back to 32 GiB if detection fails). Override with --memory-budget 512MiB / 2GiB (raw byte counts are also accepted). On GPU, a budget โ‰ค 8 GiB also shrinks the dispatch output-pool / bind-group caches (16 MiB / 128 entries). Use --fit-memory / --fit-memory 6GiB to auto-reduce accumulation, checkpointing, batch/sequence length, AMP, and model size until the plan fits. --dry-run / --plan-only prints the resolved plan and exits without tokenization or weight allocation. Budget refusals include concrete CLI/JSON remediation hints and point at the low-RAM recipe configs.

Generate text

cargo run --release --bin oxidelm -- \
  --generate model.bin "Rust learns" \
  --tokens 128

Use the complete sampling controls:

cargo run --release --bin oxidelm -- \
  --generate model.bin "Once upon a time" \
  --tokens 128 \
  --temperature 0.8 \
  --top-k 40 \
  --top-p 0.95 \
  --repetition-penalty 1.1 \
  --repetition-penalty-window 64 \
  --frequency-penalty 0.2 \
  --presence-penalty 0.1 \
  --seed 42 \
  --gpu

Bytes are decoded and streamed to standard output as tokens are produced. Generation ends at the requested token count or the tokenizer's EOS token.

For multi-prompt serving, ContinuousBatcher (and Generator::generate_batch) batches equal-position decode steps, retires finished requests, and admits waiting prompts into free slots.

For speculative decoding, pair a target model with a smaller draft via Generator::with_draft / SpeculativeGenerator. The draft proposes ฮณ tokens; the target verifies them in one cached forward and accepts a prefix with Leviathan-style rejection sampling (greedy when temperature == 0).

Inspect a checkpoint

Print parameter counts, tensor shapes, estimated FP32 memory (parameters, activations, cache, staging, optimizer), tokenizer statistics, and checkpoint metadata without loading training data:

cargo run --release --bin oxidelm -- --inspect model.bin

Analyze a model (machine-consumable)

Emit a versioned JSON contract (schema_version) covering identity (SHA-256, dtype histogram), architecture, parameter/module inventory, storage accounting, inference/training memory presets, tensor health, quantization, PEFT, tokenizer, device probe, and quality gates (fixed-seed forward validation; optional held-out perplexity / KL vs --reference). Supports OxideLM checkpoints, SafeTensors, and HF Llama-style directories. Human text is printed by default; write JSON with --json. CI uses this JSON as a release artifact gate (examples/check_analyze_gate.rs, .github/workflows/analyze-gate.yml):

cargo run --release --bin oxidelm -- \
  --analyze model.bin \
  --format auto \
  --json report.json \
  --fail-on nan,inf,schema,forward,quality \
  --corpus heldout.txt \
  --eval-corpus heldout.txt \
  --reference model.bin \
  --eval-sequence-length 32 \
  --max-kl 0.05 \
  --probe-device

Diff two checkpoints for architecture deltas, per-tensor L2, and tokenizer drift:

cargo run --release --bin oxidelm -- \
  --analyze-diff model_a.bin model_b.bin \
  --json diff.json

Quantize a checkpoint

Pack Linear weights into an inference-only checkpoint (OXLMCKP2 + quantization metadata). Default method is round-to-nearest (RTN); GPTQ / AWQ / SmoothQuant need --calibration. Embeddings and lm_head stay higher precision unless overridden. Optional activation int8 and KV-cache packs persist in schema v3 metadata and restore on load.

Weight-only RTN (int8 / int4 / NF4 / FP8):

cargo run --release --bin oxidelm -- \
  --quantize model.bin model.int8.bin \
  --method rtn \
  --bits 8 \
  --exclude embed,lm_head \
  --json quantize-report.json
cargo run --release --bin oxidelm -- \
  --quantize model.bin model.int4.bin \
  --method rtn \
  --bits 4 \
  --group-size 64 \
  --scale-method absmax
cargo run --release --bin oxidelm -- \
  --quantize model.bin model.nf4.bin \
  --method rtn \
  --format nf4

Calibration PTQ (GPTQ / AWQ / SmoothQuant):

cargo run --release --bin oxidelm -- \
  --quantize model.bin model.gptq.bin \
  --method gptq \
  --bits 4 \
  --calibration heldout.txt \
  --calibration-sequences 128 \
  --calibration-seed 0 \
  --eval-corpus heldout.txt \
  --max-ppl-delta 2.0 \
  --max-kl 0.05

Calibration-light / high-compression (HQQ, SpQR, QuIP#, AQLM):

cargo run --release --bin oxidelm -- \
  --quantize model.bin model.hqq.bin \
  --method hqq --bits 4 --group-size 64 --asymmetric

cargo run --release --bin oxidelm -- \
  --quantize model.bin model.spqr.bin --method spqr --bits 4 --group-size 32

GGUF interop (Q4_0 / Q4_K / Q5_K / Q6_K / Q8_0):

cargo run --release --bin oxidelm -- \
  --export-gguf model.bin model.Q4_K.gguf --type Q4_K

cargo run --release --bin oxidelm -- \
  --import-gguf model.Q4_K.gguf model.from-gguf.bin

Quantized SafeTensors ({module}.weight.qpacked / .qscales / .qzeros):

cargo run --release --bin oxidelm -- \
  --export-quantized model.int4.bin model.quant.safetensors

cargo run --release --bin oxidelm -- \
  --import-quantized model.quant.safetensors model.roundtrip.bin

Inference modes (weights + optional activations):

# int8 weights ร— int8 activations (dynamic per-token)
cargo run --release --bin oxidelm -- \
  --quantize model.bin model.w8a8.bin \
  --method rtn \
  --inference-mode W8A8

# int4 weights ร— int8 activations
cargo run --release --bin oxidelm -- \
  --quantize model.bin model.w4a8.bin \
  --method rtn \
  --inference-mode W4A8

# int4 weights, FP32/FP16 activations (weight-only)
cargo run --release --bin oxidelm -- \
  --quantize model.bin model.w4a16.bin \
  --method rtn \
  --inference-mode W4A16

Static activation ranges (offline collection via --calibration):

cargo run --release --bin oxidelm -- \
  --quantize model.bin model.w8a8-static.bin \
  --method rtn \
  --inference-mode W8A8 \
  --activation-mode static \
  --activation-granularity tensor \
  --calibration heldout.txt

KV-cache quantization (per-head or per-layer scales):

cargo run --release --bin oxidelm -- \
  --quantize model.bin model.w8a8-kv.bin \
  --method rtn \
  --inference-mode W8A8 \
  --kv-format int8 \
  --kv-granularity head
cargo run --release --bin oxidelm -- \
  --quantize model.bin model.kv-nf4.bin \
  --method rtn \
  --bits 8 \
  --kv-format nf4 \
  --kv-granularity layer

Useful flags: --asymmetric, --scale-method absmax|minmax|percentile|mse, --percentile P, --include <glob>, --activation-format int8, --activation-granularity tensor|token, --activation-mode static|dynamic, --kv-format int8|nf4|fp8_e4m3|fp8_e5m2, --kv-granularity layer|head, --smoothquant-alpha A, --json out.json.

Scheme matrix

Format Bits Granularity Symmetry Typical use
int8 8 tensor / channel / group:{32,64,128} sym or asym (+ZP) Default RTN; W8A8
int4 4 tensor / channel / group:{32,64,128} sym or asym Default 4-bit; W4A8 / W4A16
int3 3 same sym or asym Aggressive RTN
int2 2 same sym or asym Extreme compression
nf4 4 same symmetric QLoRA base
fp8_e4m3 8 same symmetric FP8 pack
fp8_e5m2 8 same symmetric FP8 pack

Scale methods: absmax (default), minmax, percentile, mse. Methods: rtn, gptq, awq, smoothquant, hqq, spqr, quip, aqlm.

Quantized artifact metadata (schema v3)

Inference checkpoints embed quantization in OXLMCKP2 metadata JSON:

  • schema_version โ€” must be 1..=3 (QUANTIZATION_SCHEMA_VERSION); 0 or future versions are rejected
  • method โ€” rtn|gptq|awq|smoothquant|hqq|spqr|quip|aqlm (also qat / exported from those paths)
  • bits, format, group_size / granularity, symmetric, scale_method
  • layers[] โ€” per Linear: name, format, packed/scales/(optional) zero-point tensor names, dims
  • Optional activation (W8A8 / W4A8) and kv_cache (int8 / NF4 / FP8)

Unknown methods, formats, granularities, or scheme versions fail load/save with unsupported quantization schema version โ€ฆ / unknown quantization โ€ฆ errors.

Failure modes

Symptom Likely cause
unsupported quantization schema version N Artifact from a newer OxideLM; upgrade or re-quantize
unknown quantization method/format Corrupt metadata or unsupported scheme string
Quality gate fails (--max-ppl-delta / --max-kl) Calibration too small, bits too low, or exclude embed/lm_head
Non-finite logits after load Bad packed layout / CRC; re-run --quantize
CPUโ†”GPU mismatch in tests Kernel/packing drift; see tests/quant_correctness.rs

Correctness suite: dequant error bounds, CPUโ†”GPU bounded parity, packed-layout fuzz (cargo test --test quant_correctness). Perf per scheme: cargo run --release --example quantize_benchmark -- --device cpu --json out.json against perf/baselines/quantize_schemes.json.

Standard quality evaluation

Measure held-out perplexity and top-1 next-token accuracy on fixed non-overlapping windows, then score a small greedy-completion task suite sized for tiny Shakespeare models (exact / prefix / byte overlap). Defaults to the built-in task set; pass --tasks none for perplexity only, or a JSON array of {id, prompt, expected} objects for a custom suite:

cargo run --release --bin oxidelm -- \
  --eval model.bin heldout.txt \
  --tasks builtin \
  --sequence-length 32 \
  --batch-size 1
cargo run --release --bin oxidelm -- \
  --eval model.bin \
  --tasks tasks.json

Tokenizer quality report

Measure held-out compression ratio (bytes/token), tokens per character, unknown-token rate, vocabulary utilization, and encode/decode throughput. Accepts a checkpoint or a standalone tokenizer JSON file. Training also emits these metrics on the validation partition after BPE training:

cargo run --release --bin oxidelm -- \
  --tokenizer-report model.bin heldout.txt \
  --repeats 3

Import a common tokenizer

Convert GPT-2 vocab.json + merges.txt, or a Hugging Face tokenizer.json BPE subset (byte-level vocab/merges, optional special added_tokens), into OxideLM tokenizer JSON. Token IDs are remapped to OxideLM's byte/special/merge layout; GPT-2 regex pre-tokenization is not reproduced (raw mode):

cargo run --release --bin oxidelm -- \
  --import-tokenizer vocab.json merges.txt tokenizer.json

cargo run --release --bin oxidelm -- \
  --import-tokenizer hf_tokenizer.json tokenizer.json

Export and import weights

OxideLM checkpoints can export a named SafeTensors state dict. By default the file uses OxideLM parameter names and [in, out] linear layouts. Pass --hf for Hugging Face Llama-compatible names and [out, in] layouts, and optionally write a Llama-style config.json:

cargo run --release --bin oxidelm -- \
  --export-weights model.bin model.safetensors \
  --hf \
  --config-out config.json

Import replaces the model weights in an existing checkpoint. Tokenizer and trainer metadata are kept; AdamW moments are reset:

cargo run --release --bin oxidelm -- \
  --import-weights model.safetensors model.bin \
  --hf

The architecture must already match (same hidden size, layer count, and related dims). Grouped-query attention and non-SiLU MLPs are not supported.

Library usage

Resume a saved training run from Rust:

use oxidelm::trainer::{load_checkpoint, Trainer, TrainingBackend};

fn resume(training_text: String) -> Result<(), Box<dyn std::error::Error>> {
    let checkpoint = load_checkpoint("model.bin")?;
    let tokens = checkpoint.tokenizer.encode(&training_text);
    let mut trainer = Trainer::from_checkpoint(
        checkpoint,
        tokens,
        TrainingBackend::Cpu,
    )?;

    trainer.train_steps(100)?;
    Ok(())
}

Load a Hugging Face Llama-style SafeTensors file into an OxideLM model:

use oxidelm::model::CausalLM;
use oxidelm::state_dict::{
    from_hf_state_dict, load_hf_config, load_safetensors,
};

fn load_hf(path: &str) -> Result<CausalLM, Box<dyn std::error::Error>> {
    let hf_config = load_hf_config(format!("{path}/config.json"))?;
    let config = hf_config.to_causal_lm(0)?;
    let model = CausalLM::new(config.clone())?;
    let hf_weights = load_safetensors(format!("{path}/model.safetensors"))?;
    let state = from_hf_state_dict(&config, &hf_weights)?;
    model.load_state_dict(&state)?;
    Ok(model)
}

Verification

cargo test

# GPU kernel correctness, fallback, and allocation checks only
cargo test --test gpu_verification

# Complete CPU/GPU model logits, loss, and gradient comparison
cargo test --test full_model_correctness

# Central finite-difference checks for every differentiable operation
cargo test --test numerical_gradients

The suite checks tensor math and analytical gradients against central finite differences, including broadcasting, batched matmul, embedding, RoPE, reductions, activations, Softmax, cross-entropy, RMSNorm, and GPU-only fused operations. It also checks BPE persistence and round trips, causal masking, Transformer shape and gradient flow, CPU/GPU numerical agreement, AdamW updates, initialization statistics, sampling filters, and miniature-training loss reduction. GPU cases skip cleanly when no compatible adapter is present. The dedicated GPU suite compares WGSL matmul, RMSNorm, and Softmax against CPU references at 1e-4 tolerance for 512 ร— 512 and 1 ร— 512 ร— 4096 shapes. It also verifies CPU fallback and repeated buffer cleanup. The full-model suite uses deterministic two-layer models at (B,S) values (1,1), (1,4), (2,3), and (3,5), comparing complete logits, loss, and every parameter gradient between CPU and GPU execution.

Benchmarks & Performance

OxideLM keeps tensors in contiguous row-major buffers, memory-maps tokenizer training input, and dispatches compact WGSL kernels without a Python or external ML runtime. This reduces framework overhead and keeps host/device movement visible and measurable.

[!IMPORTANT] The figures below are reference performance targets supplied for a future 1.1B-parameter FP16 benchmark, not results reproduced from this checkout. The committed end-to-end inference harness still uses FP32 model buffers and a deterministic miniature model. FP16/BF16 matmul kernels are available through the explicit mixed-precision API, but they do not reproduce the targets below. Do not publish the reference targets as measured results until the 1.1B model configuration and raw reports are committed and independently reproduced.

Reference generation targets

Backend / hardware Quantization Batch size (B=1) Batch size (B=8) TTFT (prefill, 512 tokens)
NVIDIA RTX 4090 (wgpuโ€“Vulkan) FP16 ~185 tok/s ~620 tok/s 4.2 ms
Apple M3 Max (wgpuโ€“Metal, 40 GPU cores) FP16 ~142 tok/s ~410 tok/s 6.8 ms
x86_64 CPU (AVX-512, 16 cores) FP32 ~28 tok/s ~85 tok/s 38.0 ms

wgpu does not expose a CUDA backend. NVIDIA measurements use its Vulkan or DX12 backend; CUDA should not appear in the backend label.

Measurement protocol

The existing WGSL harness compares CPU autovectorized loops with synchronized wgpu dispatches for B=1, 4, 8 and S=128, 512, 2048. Criterion reports latency and byte throughput for each case. The inference example separates prefill and cached decode so one phase cannot hide another:

Benchmark Harness Measurement
Prefill / TTFT examples/kv_cache_benchmark.rs One prompt forward through a readable first-token selection, reported separately from decode.
Autoregressive decode examples/kv_cache_benchmark.rs Steady-state one-token cached forwards after prefill. Also checks cached and uncached logits at 1e-4.
Training phases examples/training_benchmark.rs Visible data-loader wait, forward plus cross-entropy, gradient reset plus backward, and AdamW at two fixed FP32 model shapes (--suite e2e).
Low-RAM training phases examples/training_benchmark.rs --suite low_ram Same phase split with batch_size=1, activation checkpointing ON, and documented gradient-accumulation context for limited-RAM trainer settings.
CPU matrix multiplication benches/cpu_matmul.rs Reference versus blocked-parallel decode, prefill, and batched-attention shapes.
BPE encode/decode benches/tokenizer.rs Input bytes and emitted tokens per second for encode; output bytes and consumed tokens per second for decode.

Before sampling, initialize the device, compile and cache pipelines, upload weights, and run warm-up iterations. Each timed GPU iteration must wait for GPU completion or host readback; measuring command submission alone produces an invalid result. Criterion should use a fixed sample size and Throughput, while std::time::Instant records end-to-end wall-clock latency:

use std::time::Instant;

let started = Instant::now();
let logits = model.forward(&prompt);
let first_token = sample_last_position(&logits);
let ttft = started.elapsed();

let decode_started = Instant::now();
let generated = generate_tokens(&model, first_token, generation_length);
let decode_seconds = decode_started.elapsed().as_secs_f64();
let tokens_per_second = generated.len() as f64 / decode_seconds;

The benchmark implementation should pass inputs and outputs through std::hint::black_box, use the same model weights and token sequences on every backend, and publish median plus p95 latencyโ€”not only the fastest sample.

Prefill throughput is calculated independently of TTFT:

prefill tok/s = batch_size ร— prompt_tokens / prefill_seconds
decode tok/s  = batch_size ร— generated_tokens / decode_seconds
TTFT (ms)     = elapsed through first synchronized next-token result ร— 1,000

For context-length comparisons, report 128, 512, and 2,048 tokens as separate Criterion groups. Include model dimensions, thread count, OS, Rust version, wgpu backend, adapter/driver, power mode, warm-up duration, sample count, and whether host/device transfers are included.

Component micro-benchmark targets

Component Reference target What must be timed
Zero-copy byte-level BPE encode ~12.5 MB/s (~3.2M tok/s) mmap-backed input traversal through emitted token IDs
BPE decode Report MB/s and tok/s token lookup through the completed byte output buffer
RMSNorm + RoPE <0.8 ยตs per layer synchronized kernel execution, excluding one-time pipeline creation
Attention matmul Report achieved-bandwidth percentage synchronized kernel time and bytes read/written for the exact matrix shape

Fusion measurements

These synchronized medians were measured on an Apple M1 Max with Metal using 128 rows, width 512, intermediate width 2,048, and 8 attention heads. Only faster candidates remain in model execution.

Chain Separate kernels Fused kernel Result
Residual + RMSNorm, both outputs 412 ยตs 348 ยตs kept, 15% faster
SwiGLU 472 ยตs 361 ยตs kept, 23% faster
Masked Softmax 470 ยตs 408 ยตs kept, 13% faster
RMSNorm + projection 617 ยตs 824 ยตs rejected, 34% slower
QKV projection 818 ยตs 1,480 ยตs rejected, 81% slower

Attention bandwidth utilization must be derived from measured data rather than guessed:

achieved bandwidth = bytes transferred / kernel seconds
utilization (%)    = achieved bandwidth / measured device bandwidth ร— 100

Use a device-local bandwidth benchmark for the denominator. Vendor peak memory bandwidth is useful context, but it is not a substitute for measured sustainable bandwidth with the active backend.

Reproducing results

Run the GPU kernel harness with:

cargo bench --bench wgsl_bench

Filter to the shape-selection comparison with:

cargo bench --bench wgsl_bench -- wgsl_matmul_kernel_selection

Run retained fusion comparisons with:

cargo bench --bench wgsl_bench -- wgsl_fusion_candidates

Compare the reference and blocked-parallel CPU matmul backends with:

cargo bench --bench cpu_matmul

The harness uploads inputs before timing. wgpu_dispatch_sync samples include compute dispatch and synchronization. On adapters with timestamp queries, Criterion also reports wgpu_kernel_timestamp, wgpu_host_submission, and wgpu_synchronization samples for the same operations. Throughput comes from the logical buffer bytes read and written. Without a compatible adapter, CPU measurements still run and GPU measurements are omitted.

Run the deterministic CPU and GPU KV-cache correctness and inference benchmark with:

cargo run --release --example kv_cache_benchmark

The command reports prefill and decode throughput, median and p95 time to first token, peak logical tensor-buffer memory, and host/device transfer bytes per generation. GPU work is synchronized through output readback. If no compatible adapter exists, the default all mode reports CPU results and skips GPU. Select one backend or save machine-readable output with:

cargo run --release --example kv_cache_benchmark -- --device cpu --json results.json
cargo run --release --example kv_cache_benchmark -- --device gpu --json results.json

JSON output includes a metadata object with commit, Rust version, OS/arch, CPU and Rayon thread counts, adapter/driver/backend, power mode, warm-up and sample counts, transfer policy, and the command line. Commit comes from OXIDELM_COMMIT, GITHUB_SHA, or git rev-parse HEAD.

Run the fixed-shape training benchmark with:

cargo run --release --example training_benchmark
cargo run --release --example training_benchmark -- --device cpu --json training.json
cargo run --release --example training_benchmark -- --suite low_ram --device cpu --json training_low_ram.json

It reports median and p95 wall-clock time separately for data loading, forward, backward, and AdamW. The default --suite e2e runs the tiny and small shapes. --suite low_ram runs one tiny shape with batch_size=1 and activation checkpointing enabled in the timed forward/backward path. GPU phase medians are also split into device kernel time, host submission time, and host synchronization time. Kernel time comes from compute-pass timestamp queries when the adapter supports them; otherwise it is reported as unavailable. The tiny and small shapes, warm-up count, sample count, prefetch depth, seed, and FP32 precision are constants in the harness. Data-loading time measures the wait visible to the training loop after background prefetch. Low-RAM phase thresholds live in perf/baselines/training_low_ram.json; end-to-end training thresholds are in perf/baselines/training_e2e.json. Per-scheme quantized inference tokens/s and peak memory live in perf/baselines/quantize_schemes.json (harness: examples/quantize_benchmark.rs).

Profile logical tensor allocation during cached prefill and decode:

# Allocation profiling during decoding (CPU by default, GPU when available)
cargo run --release --example profile_alloc
cargo run --release --example profile_alloc -- --device cpu --json profile_alloc.json

The helper resets peak counters, runs one prefill plus a short decode loop on a tiny FP32 model, and reports peak and live tensor bytes. On CPU these come from cpu_memory_stats(); on GPU they come from GpuContext::memory_stats(). Use --device all|cpu|gpu to select backends and --json path to write machine-readable results.

A valid result submission should include criterion/ HTML or machine-readable output, the exact commit hash, complete hardware metadata, and the command line. Compare only equivalent precision, batch size, context length, transfer policy, and synchronization boundaries.

Scope

OxideLM is an educational systems project, not a replacement for mature distributed-training frameworks. It prioritizes explicit mechanics, compact safe-Rust implementations, and measured selective fusion over mixed precision, quantization, distributed execution, and production model compatibility.

Contributing

Focused improvements are welcome. Good contribution areas include profiled kernel optimization, more tensor views, checkpoint compatibility, mixed precision, and reproducible benchmarks. Include correctness tests and before/after measurements for performance changes.

License

OxideLM is offered under either of the following licenses, at your option:

Unless explicitly stated otherwise, contributions intentionally submitted for inclusion in OxideLM are dual-licensed under the same terms.

Acknowledgments

OxideLM draws inspiration from the GPT and Llama decoder families, the original Transformer and BPE literature, and the Rust, Rayon, and wgpu ecosystems. The implementation remains independent and is intended to make the underlying systems ideas easier to inspect, modify, and understand.