OxideLM ๐ฆ
A bare-metal, pure-Rust Transformer engine featuring custom reverse-mode autograd, zero-copy BPE tokenization, and cross-platform wgpu acceleration.
No Python. No PyTorch bindings. No hidden tensor runtime.
OxideLM is a from-scratch causal language-model stack written in safe Rust. It implements the path from raw UTF-8 text to generated tokens: byte-level BPE, row-major tensors, reverse-mode automatic differentiation, a decoder-only Transformer, GPU compute kernels, AdamW training, checkpointing, and sampling.
The project is designed for learning and experimentation at the systems layer. Every important mechanismโstrides, graph traversal, gradient accumulation, causal attention, normalization, optimization, and GPU dispatchโis visible in the source.
[!NOTE] โZero dependencyโ means zero Python and external ML-framework dependencies. OxideLM uses a small set of Rust infrastructure crates:
wgpu,bytemuck,pollster,memmap2,rayon,serde,serde_json,safetensors, andrand.
Architecture
Raw UTF-8 text
โ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Byte-level BPE tokenizer โ mmap input ยท parallel pair counts
โ memmap2 + rayon โ reversible byte vocabulary
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ
โ token IDs [B, S]
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Autograd tensor engine โ row-major Vec<f32> ยท dynamic strides
โ reverse topological backward โ lazy, accumulated gradients
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ
โ tensors
โโโโโโโโโโดโโโโโโโโโโ
โ โ
โผ โผ
โโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโ
โ CPU โ โ WGSL GPU kernels โ tiled matmul ยท activations
โ safe Rust โ โ wgpu โ softmax ยท RMSNorm
โโโโโโโฌโโโโโโ โโโโโโโโโโโฌโโโโโโโโโโ
โโโโโโโโโโโโฌโโโโโโโโโโ
โผ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Decoder-only CausalLM โ
โ Embedding โ N ร Block โ Norm โ
โ โ language-model head โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโ
โ logits [B, S, V]
โโโโโโโโโดโโโโโโโโโ
โผ โผ
Cross-entropy Top-k / top-p
โ backward โ decoded text
โ AdamW
Each pre-normalized Transformer block follows:
x โโ RMSNorm โโ causal attention โโ + โโ RMSNorm โโ SwiGLU โโ + โโ output
โ โฒ โฒ
โโโโโโโโโโโโโ residual โโโโโโโโโโโโโโโโโ โ
residual โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Attention projects [B, S, D] into [B, H, S, D/H], applies RoPE to queries
and keys, masks future positions, performs scaled dot-product attention, and
joins the heads back into [B, S, D].
๐ Glossary of technical terms
Why OxideLM?
| Area | What OxideLM provides |
|---|---|
| ๐งฎ Explicit autograd | Reverse-mode differentiation with topological graph traversal, on-demand gradient storage, and correct accumulation through branched graphs. |
| โก Memory-conscious tokenization | Training data is memory-mapped instead of copied wholesale. Pair-frequency work is distributed across CPU cores with Rayon. |
| ๐ง Modern decoder design | RMSNorm, rotary position embeddings, grouped-query causal attention, SwiGLU, configurable token-embedding/LM-head weight tying, and cross-entropy loss. |
| ๐ข Integer token paths | IndexTensor keeps token IDs exact as CPU integers and uploads typed u32 index buffers for GPU embedding and loss kernels. |
| ๐ฅ๏ธ Parallel CPU matmul | Safe blocked matrix multiplication distributes output rows with Rayon and retains the original triple loop as a selectable reference. |
| ๐ฎ Portable GPU compute | Persistent parameter buffers and device-resident WGSL matmul, fused residual + RMSNorm, SwiGLU, masked Softmax, and activation kernels. |
| ๐ฆ Quantization | Weight-only RTN/GPTQ/AWQ/SmoothQuant/HQQ/SpQR/QuIP#/AQLM (int8โint2, NF4, FP8), mixed modes W8A8 / W4A8 / W4A16, KV-cache int8/FP8/NF4, plus GGUF + quantized SafeTensors interop. |
| ๐ก๏ธ Safe systems code | No project-authored unsafe; ownership and lifetimes remain explicit across tensor graphs and GPU resources. |
| ๐ฌ Readable internals | Compact modules expose the mechanics usually hidden behind Python APIs and mature ML frameworks. |
| ๐๏ธ End-to-end training | Depth-scaled Transformer initialization, gradient clipping, cosine warmup scheduling, AdamW, resumable checkpoints, and throughput reporting. |
| โ๏ธ Practical decoding | Per-layer KV caching, one-token decode steps, temperature, top-k, nucleus sampling, repetition / frequency / presence penalties, seeded randomness, EOS stopping, continuous batching, and speculative decoding with a smaller draft model. |
CPU matrix multiplication
Tensor::matmul uses CpuMatmulBackend::BlockedParallel by default. It blocks
the shared and output-column dimensions, distributes independent output rows
with Rayon, and falls back to the reference loop when a matrix has too few rows
to benefit from threads. Select either implementation globally or per call:
use ;
set_cpu_matmul_backend;
let reference = left.matmul;
let optimized = left.matmul_with_backend;
GPU execution
Host Vec<f32>
โ to_gpu() returns GPU Tensor
โ GPU storage buffer shared by reshape views
โ device-resident kernel chain
โ to_cpu() requests readback
โ pooled readback staging buffer
Matrix multiplication ships a safe 16 ร 16 tiled kernel, an
alignment-specialized 16 ร 16 kernel without boundary branches, a benchmark
candidate using 32 ร 32 register blocking, and a vec4-loaded 8 ร 32 kernel.
On the first use of a matrix shape, automatic selection times every valid tile
and workgroup configuration. The fastest median is cached by adapter identity,
driver, backend, and matrix dimensions. Timestamp-query kernel time is used
when available; synchronized wall time is the fallback. Element-wise and
row-reduction kernels use 256 threads per workgroup. Mixed-precision matrix
operands can use FP16 or packed BF16 storage while multiplying and accumulating
in FP32. FP16 is enabled only when the adapter exposes SHADER_F16; otherwise
the upload selects the FP32 kernel. Packed BF16 uses portable core WGSL bit
operations.
wgpu targets Metal, Vulkan, DX12, and browser WebGPU; NVIDIA hardware is
reached through Vulkan or DX12 rather than a CUDA-specific backend.
Model parameters receive persistent device buffers through prepare_gpu, which
forward_gpu calls automatically. Parameter updates invalidate those buffers,
so the next GPU forward uploads the new values once. GPU operations do not read
results back automatically. Tensor::data(), item(), and CPU math accept only
CPU-resident tensors; call Tensor::to_cpu() at the wanted readback boundary.
Likewise, Tensor::to_gpu() returns a GPU-resident tensor and reuses cached
uploads. Multi-head attentionโincluding head permutation, RoPE, tiled K/V
reads, online causal Softmax, value aggregation, and KV-cache growthโstays on
the GPU without materializing a quadratic score matrix.
Linear weights can be quantized for inference. Weight-only paths keep activations
in FP32; mixed modes (W8A8, W4A8) also quantize activations to int8
(static or dynamic, per-tensor or per-token). Optional KV-cache quantization
packs keys/values as int8, FP8, or NF4 with per-head or per-layer scales:
use WeightQuantization;
model.quantize_linear_weights;
Int8 packs four weights per u32; Int4 packs eight. Each logical output
channel has its own symmetric scale. GPU preparation uploads the packed weights
and scales instead of the corresponding FP32 Linear buffers. The original FP32
parameters remain on the CPU for checkpoints and can be restored as the active
inference path with clear_quantized_linear_weights(). For the product CLI
(--quantize, inference modes, calibration, KV packs), see
Quantize a checkpoint.
For quantization-aware training (QAT), enable fake-quant + STE while keeping FP32/BF16
masters trainable (compatible with --mixed-precision):
Learnable scales (LSQ): --qat-learnable-scales or "learnable_scales": true in the
trainer qat object. After training, pack with oxidelm --export-qat model.bin out.bin.
QLoRA freezes an NF4/int4 base and trains LoRA/DoRA adapters:
# in training JSON under trainer.lora:
# { "rank": 8, "alpha": 16, "base_quant": "nf4", "base_group_size": 64, "targets": ["q_proj","v_proj"] }
Token IDs and cross-entropy targets use IndexTensor, not floating-point
Tensor. CPU embedding and loss code index integers directly. GPU execution
checks the WebGPU u32 limit during the first upload, caches that typed buffer,
and has no float-to-index conversion inside WGSL kernels.
Set CausalLmConfig::tie_word_embeddings to share the [vocab, hidden] token
embedding with the transposed LM projection. Tied mode exposes the shared tensor
once through parameters(), so SGD, AdamW, GPU AdamW, and checkpoints update or
store it once. Older checkpoints without this field load in untied mode.
CausalLmConfig::initializer_std controls the normal distribution used for
model weights and defaults to 0.02. Attention output and FFN down projections
use initializer_std / sqrt(2 * num_layers) to keep residual activations and
gradients stable as depth grows. Training JSON accepts model.initializer_std;
the CLI override is --initializer-std.
Set CausalLmConfig::num_key_value_heads below num_heads to share each K/V
head across a group of query heads. The query-head count must be divisible by
the K/V-head count. K/V projections and caches use only
num_key_value_heads * head_dim; training JSON and CLI use
model.num_key_value_heads and --num-key-value-heads.
Attention score and value products use explicit batched GEMM:
[B, H, M, K] ร [B, H, K, N] -> [B, H, M, N]. The B * H matrices share
one tiled GPU dispatch and never enter the CPU matmul path.
GPU training uses device-side cross-entropy and backward_gpu() kernels for
matmul, RMSNorm, Softmax, SwiGLU, embeddings, RoPE, masking, permutation, and
residual accumulation. AdamW parameters, gradients, and first/second moments
remain in GPU buffers across steps. Only scalar loss and gradient-norm metrics
are read back; checkpoint saving explicitly snapshots weights and moments to
CPU storage.
Repeated dispatches reuse output storage from a shape-keyed LRU pool and reuse
bind groups when the pipeline, buffers, and uniform parameters match. Defaults
bound pooled outputs to 64 MiB and bind groups to 512 entries. A CLI
--memory-budget of 8 GiB or less selects smaller defaults (16 MiB / 128
entries) via GpuContext::apply_memory_budget โ set_cache_limits. Use
set_cache_limits to change the bounds directly, trim_dispatch_caches to
release them, and memory_stats to inspect allocations, reuse, cache hits,
and retained bytes.
Check buffer residency and transfer counters on a machine with a GPU:
Project layout
OxideLM/
โโโ Cargo.toml
โโโ Cargo.lock
โโโ benches/
โ โโโ cpu_matmul.rs # reference vs blocked-parallel matmul
โ โโโ wgsl_bench.rs # WGSL kernel harness and fusion comparisons
โโโ examples/
โ โโโ gpu_residency.rs # buffer residency and transfer counters
โ โโโ kv_cache_benchmark.rs # prefill / decode benchmark with KV-cache
โ โโโ training_benchmark.rs # fixed-shape training phase timings
โโโ shaders/
โ โโโ matmul.wgsl # tiled matrix multiplication
โ โโโ matmul_vec4.wgsl # vec4-loaded 8ร32 matmul variant
โ โโโ matmul_fp16.wgsl # FP16 product, FP32 accumulation
โ โโโ matmul_bf16.wgsl # packed BF16 input, FP32 accumulation
โ โโโ matmul_quantized.wgsl # packed int8/int4 Linear inference
โ โโโ fusion.wgsl # measured Transformer operation fusions
โ โโโ index.wgsl # integer embedding and loss kernels
โ โโโ elementwise.wgsl # add, activations, softmax, RMSNorm
โ โโโ attention.wgsl # RoPE, permutations, causal masking
โ โโโ tiled_attention.wgsl # online-Softmax tiled attention + backward
โ โโโ backward.wgsl # backpropagation kernels (matmul, RMSNorm, etc.)
โ โโโ optimizer.wgsl # GPU-resident AdamW step
โโโ src/
โ โโโ lib.rs # tensor storage, operations, autograd
โ โโโ tokenizer.rs # mmap + parallel byte-level BPE
โ โโโ tokenizer_quality.rs # held-out compression / coverage / throughput reports
โ โโโ quality.rs # held-out perplexity / next-token accuracy / tiny task suite
โ โโโ layers.rs # linear and embedding layers
โ โโโ norm.rs # RMSNorm
โ โโโ attention.rs # RoPE and causal multi-head attention
โ โโโ transformer.rs # SwiGLU and Transformer blocks
โ โโโ model.rs # decoder-only CausalLM
โ โโโ gpu.rs # wgpu runtime and graph integration
โ โโโ initialization.rs # normal, Xavier, and Kaiming initialization
โ โโโ optimizer.rs # AdamW
โ โโโ data_loader.rs # mmap token shards and batch prefetch
โ โโโ data_parallel.rs # CPU data-parallel replica pool + grad reduce
โ โโโ corpus.rs # .txt / JSONL / directory corpus discovery + concat
โ โโโ trainer.rs # batching, schedules, training loop
โ โโโ checkpoint.rs # binary named atomic checkpoints
โ โโโ log.rs # quiet / human / JSON structured logging
โ โโโ profile.rs # named timing spans and aggregate profile reports
โ โโโ memory.rs # FP32 memory estimates and device-budget checks
โ โโโ config.rs # train JSON configs, CLI overrides, pre-alloc validation
โ โโโ inspect.rs # checkpoint parameter / memory / tokenizer reports
โ โโโ analyze.rs # versioned --analyze JSON report (ops / CI)
โ โโโ weight_quant.rs # RTN weight packs (int/NF4/FP8)
โ โโโ act_quant.rs # activation int8 + W8A8/W4A8/W4A16 modes
โ โโโ kv_quant.rs # KV-cache int8/FP8/NF4 packs
โ โโโ calibration.rs # calibration corpus / token-shard loader
โ โโโ ptq.rs # GPTQ / AWQ / SmoothQuant
โ โโโ advanced_quant.rs # HQQ / SpQR / QuIP# / AQLM
โ โโโ gguf.rs # GGUF Q4_0/Q4_K/Q5_K/Q6_K/Q8_0 I/O
โ โโโ quantize.rs # --quantize product surface + reports
โ โโโ state_dict.rs # named state dicts, SafeTensors, HF Llama + quantized packs
โ โโโ generate.rs # sampling, autoregressive / speculative decoding, continuous batching
โ โโโ main.rs # train / generate / inspect / analyze / quantize / eval / tokenizer / weight I/O CLI
โโโ tests/
โ โโโ tensor_engine.rs # tensor math and analytical gradient checks
โ โโโ tokenizer.rs # BPE persistence and round-trip tests
โ โโโ causal_lm.rs # masking, shape, and gradient-flow tests
โ โโโ gpu.rs # WGSL vs CPU numerical agreement checks
โ โโโ gpu_verification.rs # dedicated GPU kernel correctness suite
โ โโโ full_model_correctness.rs # complete CPU/GPU logits, loss, and gradient comparison
โ โโโ numerical_gradients.rs # central finite-difference checks for every op
โ โโโ property_fuzz.rs # proptest: shapes, tokenizer, checkpoints, sampling, GPU errors
โโโ training/
โโโ input.txt # training corpus (sample text; also .jsonl or a directory)
โโโ tiny.json # smoke / CI miniature config
โโโ low-ram-8gb.json # 8 GiB RAM recipe (checkpointing + AMP bf16)
โโโ cpu-laptop.json # conservative CPU-laptop recipe
โโโ gpu-16gb.json # larger GPU recipe under ~16 GiB
โโโ schema.json # JSON Schema for TrainConfigFile (`--train init --schema`)
โโโ model.bin # saved model checkpoint
โโโ model.best.bin # best validation checkpoint
Quick start
Prerequisites
- A current stable Rust toolchain with Cargo
- A GPU and current graphics driver only when using
--gpu
Feature flags (lean builds)
OxideLM ships two opt-out Cargo features, both enabled by default:
| Feature | Enables | Pulls in |
|---|---|---|
gpu |
--gpu, forward_gpu/*_gpu methods, wgpu-backed tensors |
wgpu, pollster |
training |
--train, Trainer, data loading, data-parallel, config resolution |
(no extra deps) |
On constrained machines (no GPU, inference-only), build without either:
This compiles a lean CPU tensor/tokenizer/--generate binary with no wgpu in the
dependency tree. --train and --gpu exit with a clear error telling you which
feature to rebuild with instead of failing to compile.
For CPU-only training (no GPU driver/wgpu required):
The default cargo build / cargo build --release keeps today's full behavior
(gpu + training both enabled).
Train a miniature model
Generate a starter config from an architecture preset and memory budget (fits knobs automatically; default preset llama-tiny, default budget ~70% of host RAM). Presets: llama-tiny / gpt-tiny (smoke), llama-micro / gpt-micro (โค8 GiB FP32 + checkpointing), llama-small / gpt-small:
Write the JSON Schema for config files:
Smoke train from the repo root (paths are relative to the current working directory):
Train on ~8 GiB RAM:
Train on a CPU laptop:
Preview a memory plan without tokenizing or allocating weights:
Override dimensions on the CLI (CLI flags win over the config file):
Data-parallel training (--data-parallel-workers N, N > 1) splits each
micro-batch across N replica threads, mean-reduces gradients, then steps the
primary optimizer. Batch size must be divisible by N. On --gpu, workers use
distinct adapters when available, otherwise multiple logical devices on one
adapter (GpuContext::create_devices).
Effective batch size. One optimizer step consumes
batch_size ร gradient_accumulation_steps sequences (logged as
effective_batch_size in the train / dry-run plan). The three knobs interact
as follows:
| Knob | Role |
|---|---|
batch_size (--batch-size) |
Micro-batch width: sequences per forward/backward. Peak activation memory scales with this. |
gradient_accumulation_steps (--gradient-accumulation-steps) |
Micro-batches per optimizer step. Gradients are scaled by 1/steps and summed before AdamW. Multiplies effective batch without growing micro-batch activations. |
data_parallel_workers (--data-parallel-workers) |
Shards each micro-batch across N replicas (batch_size must be โฅ N and divisible by N). Speeds the step; does not change effective batch size. |
Example: batch_size=4, gradient_accumulation_steps=8, data_parallel_workers=2
โ effective batch 32; each of two workers sees shard size 2 per micro-batch;
eight micro-batches run before one optimizer step. Raise accumulation (not
workers) when you want a larger statistical batch under a tight RAM budget.
See also Effective Batch Size in the glossary.
Enable GPU-accelerated projections and normalization:
Resume from a checkpoint (restores weights, AdamW moments, AMP loss scaler,
activation-checkpoint interval, offload settings, schedule, sampler cursor,
tokenizer, and model seed). --steps / --max-steps are aliases for the
session step count (additional optimizer steps on resume). JSON
schedule.total_steps is the LR schedule horizon and the default session length
when the CLI omits them; if resume would pass that horizon, OxideLM extends it
and prints a clear steps log line. When CLI and config disagree, the CLI wins
for the session and a clarifying message is logged. Omit low-RAM CLI/JSON flags
on resume, or keep them identical to the checkpoint โ mismatches error:
Run validation evaluation without further optimizer steps (--eval-only; often
with --resume). Updates the best checkpoint when validation improves; leaves
the final model.bin unchanged:
Training accepts a single .txt file, a JSONL file of {"text":โฆ} records, or a
directory of .txt / .jsonl files. Directories and multi-file / JSONL inputs are
concatenated in sorted path order into the token cache (*.tokens/source/) so
token sharding stays deterministic across runs. To continue on another corpus with
the same checkpoint, pass the new input, keep --resume, and rebuild the token
cache (the tokenizer stays the one stored in the checkpoint):
Examples:
# directory of text shards
# JSONL documents
The trainer packs the corpus into non-overlapping sequences and shuffles them
once per epoch. Shuffling is derived from the model seed, so runs and checkpoint
resumes are reproducible. Attention, residual, and embedding dropout use
step-derived deterministic masks during training and are disabled for
evaluation and generation. Raw corpus bytes and encoded token shards are
memory-mapped. Tokenization writes bounded chunks to model.bin.tokens/ by
default, and a bounded background queue prepares upcoming batches. Override the
cache with --token-cache, or tune data.shard_tokens,
data.tokenize_chunk_bytes, and data.prefetch_batches in JSON (the matching
CLI flags are --shard-tokens, --tokenize-chunk-bytes, and
--prefetch-batches). The trainer then runs:
Long-context models can enable Hugging Face-style linear or YaRN RoPE scaling
in the JSON model section. max_sequence_length must not exceed the original
training context multiplied by factor:
Use "linear" for rope_type to select linear position interpolation. YaRN
also accepts optional attention_factor, beta_fast, and beta_slow values.
Omitting rope_scaling keeps the original short-context RoPE behavior.
zero gradients โ forward โ cross-entropy โ backward
โ clip gradients โ AdamW step โ checkpoint
The final part of the corpus is held out before tokenizer training. Evaluation
uses fixed, non-overlapping validation sequences and never performs backward or
optimizer steps. Logs report validation loss, perplexity, validation and
training tokens processed, wall time, and the best checkpoint. By default the
best model is saved beside model.bin as model.best.bin; override it with
--best-checkpoint path.bin. Early stopping and best-checkpoint retention use
validation loss only: set data.early_stopping_patience / --early-stopping-patience
to stop after that many non-improving evaluations (0 disables), and optionally
data.early_stopping_min_delta / --early-stopping-min-delta for the required
decrease.
Training logs also report loss, perplexity, learning rate, gradient norm,
tokens per second, step ETA, tokens seen, best validation loss (after the first
eval), peak process RSS, and GPU live bytes when training on GPU. Choose output
shape with --log human (default), --log json, or --log quiet. For external
plotting, write one JSON object per optimizer step with
--metrics-jsonl metrics.jsonl (or trainer.metrics_jsonl in the config): fields
are step, loss, ppl, lr, grad_norm, loss_scale, and rss (peak RSS
bytes, or null when unavailable). Library code emits structured events through
oxidelm::log instead of hard-coded println! calls. Checkpoints are binary (OXLMCKP2): named tensors
with shapes/dtypes/CRC32 checksums, optimizer moments, tokenizer state, schedule state,
RNG and data-loader state, trainer position, and best validation loss. Saves write a
temporary file, validate it, then rename.
Enable timing spans with --profile (summary) or --profile trace (each span plus
summary). Equivalent env: OXIDELM_PROFILE=summary|trace. Named spans cover
tokenizer encode/decode/train, data-batch loading, each transformer layer
(layer.{i}), GPU upload/dispatch/synchronize, sampling, and checkpoint I/O.
When profiling is off the guards are near zero cost; on exit the CLI flushes
aggregated count / total / mean / max milliseconds per span.
Before allocating model weights, training resolves the config file and CLI flags,
validates architecture / schedule / optimizer settings, prints an FP32 memory plan
(parameters, activations, KV cache, staging, optimizer), and refuses configs that
obviously exceed the selected device budget. When --memory-budget is omitted on
CPU, the soft total defaults to about 70% of detected usable host RAM (falling
back to 32 GiB if detection fails). Override with --memory-budget 512MiB /
2GiB (raw byte counts are also accepted). On GPU, a budget โค 8 GiB also shrinks
the dispatch output-pool / bind-group caches (16 MiB / 128 entries). Use
--fit-memory / --fit-memory 6GiB
to auto-reduce accumulation, checkpointing, batch/sequence length, AMP, and model
size until the plan fits. --dry-run / --plan-only prints the resolved plan and
exits without tokenization or weight allocation. Budget refusals include concrete
CLI/JSON remediation hints and point at the low-RAM recipe configs.
Generate text
Use the complete sampling controls:
Bytes are decoded and streamed to standard output as tokens are produced. Generation ends at the requested token count or the tokenizer's EOS token.
For multi-prompt serving, ContinuousBatcher (and Generator::generate_batch)
batches equal-position decode steps, retires finished requests, and admits
waiting prompts into free slots.
For speculative decoding, pair a target model with a smaller draft via
Generator::with_draft / SpeculativeGenerator. The draft proposes ฮณ tokens;
the target verifies them in one cached forward and accepts a prefix with
Leviathan-style rejection sampling (greedy when temperature == 0).
Inspect a checkpoint
Print parameter counts, tensor shapes, estimated FP32 memory (parameters, activations, cache, staging, optimizer), tokenizer statistics, and checkpoint metadata without loading training data:
Analyze a model (machine-consumable)
Emit a versioned JSON contract (schema_version) covering identity (SHA-256,
dtype histogram), architecture, parameter/module inventory, storage accounting,
inference/training memory presets, tensor health, quantization, PEFT, tokenizer,
device probe, and quality gates (fixed-seed forward validation; optional
held-out perplexity / KL vs --reference). Supports OxideLM checkpoints,
SafeTensors, and HF Llama-style directories. Human text is printed by default;
write JSON with --json. CI uses this JSON as a release artifact gate
(examples/check_analyze_gate.rs, .github/workflows/analyze-gate.yml):
Diff two checkpoints for architecture deltas, per-tensor L2, and tokenizer drift:
Quantize a checkpoint
Pack Linear weights into an inference-only checkpoint (OXLMCKP2 + quantization
metadata). Default method is round-to-nearest (RTN); GPTQ / AWQ / SmoothQuant
need --calibration. Embeddings and lm_head stay higher precision unless
overridden. Optional activation int8 and KV-cache packs persist in schema v3
metadata and restore on load.
Weight-only RTN (int8 / int4 / NF4 / FP8):
Calibration PTQ (GPTQ / AWQ / SmoothQuant):
Calibration-light / high-compression (HQQ, SpQR, QuIP#, AQLM):
GGUF interop (Q4_0 / Q4_K / Q5_K / Q6_K / Q8_0):
Quantized SafeTensors ({module}.weight.qpacked / .qscales / .qzeros):
Inference modes (weights + optional activations):
# int8 weights ร int8 activations (dynamic per-token)
# int4 weights ร int8 activations
# int4 weights, FP32/FP16 activations (weight-only)
Static activation ranges (offline collection via --calibration):
KV-cache quantization (per-head or per-layer scales):
Useful flags: --asymmetric, --scale-method absmax|minmax|percentile|mse,
--percentile P, --include <glob>, --activation-format int8,
--activation-granularity tensor|token, --activation-mode static|dynamic,
--kv-format int8|nf4|fp8_e4m3|fp8_e5m2, --kv-granularity layer|head,
--smoothquant-alpha A, --json out.json.
Scheme matrix
| Format | Bits | Granularity | Symmetry | Typical use |
|---|---|---|---|---|
int8 |
8 | tensor / channel / group:{32,64,128} | sym or asym (+ZP) | Default RTN; W8A8 |
int4 |
4 | tensor / channel / group:{32,64,128} | sym or asym | Default 4-bit; W4A8 / W4A16 |
int3 |
3 | same | sym or asym | Aggressive RTN |
int2 |
2 | same | sym or asym | Extreme compression |
nf4 |
4 | same | symmetric | QLoRA base |
fp8_e4m3 |
8 | same | symmetric | FP8 pack |
fp8_e5m2 |
8 | same | symmetric | FP8 pack |
Scale methods: absmax (default), minmax, percentile, mse. Methods: rtn, gptq, awq, smoothquant, hqq, spqr, quip, aqlm.
Quantized artifact metadata (schema v3)
Inference checkpoints embed quantization in OXLMCKP2 metadata JSON:
schema_versionโ must be1..=3(QUANTIZATION_SCHEMA_VERSION);0or future versions are rejectedmethodโrtn|gptq|awq|smoothquant|hqq|spqr|quip|aqlm(alsoqat/exportedfrom those paths)bits,format,group_size/granularity,symmetric,scale_methodlayers[]โ per Linear:name,format, packed/scales/(optional) zero-point tensor names, dims- Optional
activation(W8A8 / W4A8) andkv_cache(int8 / NF4 / FP8)
Unknown methods, formats, granularities, or scheme versions fail load/save with
unsupported quantization schema version โฆ / unknown quantization โฆ errors.
Failure modes
| Symptom | Likely cause |
|---|---|
unsupported quantization schema version N |
Artifact from a newer OxideLM; upgrade or re-quantize |
unknown quantization method/format |
Corrupt metadata or unsupported scheme string |
Quality gate fails (--max-ppl-delta / --max-kl) |
Calibration too small, bits too low, or exclude embed/lm_head |
| Non-finite logits after load | Bad packed layout / CRC; re-run --quantize |
| CPUโGPU mismatch in tests | Kernel/packing drift; see tests/quant_correctness.rs |
Correctness suite: dequant error bounds, CPUโGPU bounded parity, packed-layout fuzz
(cargo test --test quant_correctness). Perf per scheme:
cargo run --release --example quantize_benchmark -- --device cpu --json out.json
against perf/baselines/quantize_schemes.json.
Standard quality evaluation
Measure held-out perplexity and top-1 next-token accuracy on fixed
non-overlapping windows, then score a small greedy-completion task suite sized
for tiny Shakespeare models (exact / prefix / byte overlap). Defaults to the
built-in task set; pass --tasks none for perplexity only, or a JSON array of
{id, prompt, expected} objects for a custom suite:
Tokenizer quality report
Measure held-out compression ratio (bytes/token), tokens per character, unknown-token rate, vocabulary utilization, and encode/decode throughput. Accepts a checkpoint or a standalone tokenizer JSON file. Training also emits these metrics on the validation partition after BPE training:
Import a common tokenizer
Convert GPT-2 vocab.json + merges.txt, or a Hugging Face tokenizer.json
BPE subset (byte-level vocab/merges, optional special added_tokens), into
OxideLM tokenizer JSON. Token IDs are remapped to OxideLM's byte/special/merge
layout; GPT-2 regex pre-tokenization is not reproduced (raw mode):
Export and import weights
OxideLM checkpoints can export a named SafeTensors state dict. By default the
file uses OxideLM parameter names and [in, out] linear layouts. Pass --hf
for Hugging Face Llama-compatible names and [out, in] layouts, and optionally
write a Llama-style config.json:
Import replaces the model weights in an existing checkpoint. Tokenizer and trainer metadata are kept; AdamW moments are reset:
The architecture must already match (same hidden size, layer count, and related dims). Grouped-query attention and non-SiLU MLPs are not supported.
Library usage
Resume a saved training run from Rust:
use ;
Load a Hugging Face Llama-style SafeTensors file into an OxideLM model:
use CausalLM;
use ;
Verification
# GPU kernel correctness, fallback, and allocation checks only
# Complete CPU/GPU model logits, loss, and gradient comparison
# Central finite-difference checks for every differentiable operation
The suite checks tensor math and analytical gradients against central finite
differences, including broadcasting, batched matmul, embedding, RoPE,
reductions, activations, Softmax, cross-entropy, RMSNorm, and GPU-only fused
operations. It also checks BPE persistence and round trips, causal masking,
Transformer shape and gradient flow, CPU/GPU
numerical agreement, AdamW updates, initialization statistics, sampling
filters, and miniature-training loss reduction. GPU cases skip cleanly when no
compatible adapter is present. The dedicated GPU suite compares WGSL matmul,
RMSNorm, and Softmax against CPU references at 1e-4 tolerance for 512 ร 512
and 1 ร 512 ร 4096 shapes. It also verifies CPU fallback and repeated buffer
cleanup. The full-model suite uses deterministic two-layer models at (B,S)
values (1,1), (1,4), (2,3), and (3,5), comparing complete logits, loss,
and every parameter gradient between CPU and GPU execution.
Benchmarks & Performance
OxideLM keeps tensors in contiguous row-major buffers, memory-maps tokenizer training input, and dispatches compact WGSL kernels without a Python or external ML runtime. This reduces framework overhead and keeps host/device movement visible and measurable.
[!IMPORTANT] The figures below are reference performance targets supplied for a future 1.1B-parameter FP16 benchmark, not results reproduced from this checkout. The committed end-to-end inference harness still uses FP32 model buffers and a deterministic miniature model. FP16/BF16 matmul kernels are available through the explicit mixed-precision API, but they do not reproduce the targets below. Do not publish the reference targets as measured results until the 1.1B model configuration and raw reports are committed and independently reproduced.
Reference generation targets
| Backend / hardware | Quantization | Batch size (B=1) | Batch size (B=8) | TTFT (prefill, 512 tokens) |
|---|---|---|---|---|
NVIDIA RTX 4090 (wgpuโVulkan) |
FP16 | ~185 tok/s | ~620 tok/s | 4.2 ms |
Apple M3 Max (wgpuโMetal, 40 GPU cores) |
FP16 | ~142 tok/s | ~410 tok/s | 6.8 ms |
| x86_64 CPU (AVX-512, 16 cores) | FP32 | ~28 tok/s | ~85 tok/s | 38.0 ms |
wgpu does not expose a CUDA backend. NVIDIA measurements use its Vulkan or
DX12 backend; CUDA should not appear in the backend label.
Measurement protocol
The existing WGSL harness compares CPU autovectorized loops with synchronized
wgpu dispatches for B=1, 4, 8 and S=128, 512, 2048. Criterion reports
latency and byte throughput for each case. The inference example separates
prefill and cached decode so one phase cannot hide another:
| Benchmark | Harness | Measurement |
|---|---|---|
| Prefill / TTFT | examples/kv_cache_benchmark.rs |
One prompt forward through a readable first-token selection, reported separately from decode. |
| Autoregressive decode | examples/kv_cache_benchmark.rs |
Steady-state one-token cached forwards after prefill. Also checks cached and uncached logits at 1e-4. |
| Training phases | examples/training_benchmark.rs |
Visible data-loader wait, forward plus cross-entropy, gradient reset plus backward, and AdamW at two fixed FP32 model shapes (--suite e2e). |
| Low-RAM training phases | examples/training_benchmark.rs --suite low_ram |
Same phase split with batch_size=1, activation checkpointing ON, and documented gradient-accumulation context for limited-RAM trainer settings. |
| CPU matrix multiplication | benches/cpu_matmul.rs |
Reference versus blocked-parallel decode, prefill, and batched-attention shapes. |
| BPE encode/decode | benches/tokenizer.rs |
Input bytes and emitted tokens per second for encode; output bytes and consumed tokens per second for decode. |
Before sampling, initialize the device, compile and cache pipelines, upload
weights, and run warm-up iterations. Each timed GPU iteration must wait for GPU
completion or host readback; measuring command submission alone produces an
invalid result. Criterion should use a fixed sample size and Throughput, while
std::time::Instant records end-to-end wall-clock latency:
use Instant;
let started = now;
let logits = model.forward;
let first_token = sample_last_position;
let ttft = started.elapsed;
let decode_started = now;
let generated = generate_tokens;
let decode_seconds = decode_started.elapsed.as_secs_f64;
let tokens_per_second = generated.len as f64 / decode_seconds;
The benchmark implementation should pass inputs and outputs through
std::hint::black_box, use the same model weights and token sequences on every
backend, and publish median plus p95 latencyโnot only the fastest sample.
Prefill throughput is calculated independently of TTFT:
prefill tok/s = batch_size ร prompt_tokens / prefill_seconds
decode tok/s = batch_size ร generated_tokens / decode_seconds
TTFT (ms) = elapsed through first synchronized next-token result ร 1,000
For context-length comparisons, report 128, 512, and 2,048 tokens as separate
Criterion groups. Include model dimensions, thread count, OS, Rust version,
wgpu backend, adapter/driver, power mode, warm-up duration, sample count, and
whether host/device transfers are included.
Component micro-benchmark targets
| Component | Reference target | What must be timed |
|---|---|---|
| Zero-copy byte-level BPE encode | ~12.5 MB/s (~3.2M tok/s) | mmap-backed input traversal through emitted token IDs |
| BPE decode | Report MB/s and tok/s | token lookup through the completed byte output buffer |
| RMSNorm + RoPE | <0.8 ยตs per layer | synchronized kernel execution, excluding one-time pipeline creation |
| Attention matmul | Report achieved-bandwidth percentage | synchronized kernel time and bytes read/written for the exact matrix shape |
Fusion measurements
These synchronized medians were measured on an Apple M1 Max with Metal using 128 rows, width 512, intermediate width 2,048, and 8 attention heads. Only faster candidates remain in model execution.
| Chain | Separate kernels | Fused kernel | Result |
|---|---|---|---|
| Residual + RMSNorm, both outputs | 412 ยตs | 348 ยตs | kept, 15% faster |
| SwiGLU | 472 ยตs | 361 ยตs | kept, 23% faster |
| Masked Softmax | 470 ยตs | 408 ยตs | kept, 13% faster |
| RMSNorm + projection | 617 ยตs | 824 ยตs | rejected, 34% slower |
| QKV projection | 818 ยตs | 1,480 ยตs | rejected, 81% slower |
Attention bandwidth utilization must be derived from measured data rather than guessed:
achieved bandwidth = bytes transferred / kernel seconds
utilization (%) = achieved bandwidth / measured device bandwidth ร 100
Use a device-local bandwidth benchmark for the denominator. Vendor peak memory bandwidth is useful context, but it is not a substitute for measured sustainable bandwidth with the active backend.
Reproducing results
Run the GPU kernel harness with:
Filter to the shape-selection comparison with:
Run retained fusion comparisons with:
Compare the reference and blocked-parallel CPU matmul backends with:
The harness uploads inputs before timing. wgpu_dispatch_sync samples include
compute dispatch and synchronization. On adapters with timestamp queries,
Criterion also reports wgpu_kernel_timestamp, wgpu_host_submission, and
wgpu_synchronization samples for the same operations. Throughput comes from
the logical buffer bytes read and written. Without a compatible adapter, CPU
measurements still run and GPU measurements are omitted.
Run the deterministic CPU and GPU KV-cache correctness and inference benchmark with:
The command reports prefill and decode throughput, median and p95 time to first
token, peak logical tensor-buffer memory, and host/device transfer bytes per
generation. GPU work is synchronized through output readback. If no compatible
adapter exists, the default all mode reports CPU results and skips GPU. Select
one backend or save machine-readable output with:
JSON output includes a metadata object with commit, Rust version, OS/arch, CPU
and Rayon thread counts, adapter/driver/backend, power mode, warm-up and sample
counts, transfer policy, and the command line. Commit comes from OXIDELM_COMMIT,
GITHUB_SHA, or git rev-parse HEAD.
Run the fixed-shape training benchmark with:
It reports median and p95 wall-clock time separately for data loading, forward,
backward, and AdamW. The default --suite e2e runs the tiny and small shapes.
--suite low_ram runs one tiny shape with batch_size=1 and activation
checkpointing enabled in the timed forward/backward path. GPU phase medians are also split into device kernel time,
host submission time, and host synchronization time. Kernel time comes from
compute-pass timestamp queries when the adapter supports them; otherwise it is
reported as unavailable. The tiny and small shapes, warm-up count, sample
count, prefetch depth, seed, and FP32 precision are constants in the harness.
Data-loading time measures the wait visible to the training loop after
background prefetch. Low-RAM phase thresholds live in
perf/baselines/training_low_ram.json; end-to-end training thresholds are in
perf/baselines/training_e2e.json. Per-scheme quantized inference tokens/s and
peak memory live in perf/baselines/quantize_schemes.json (harness:
examples/quantize_benchmark.rs).
Profile logical tensor allocation during cached prefill and decode:
# Allocation profiling during decoding (CPU by default, GPU when available)
The helper resets peak counters, runs one prefill plus a short decode loop on a
tiny FP32 model, and reports peak and live tensor bytes. On CPU these come from
cpu_memory_stats(); on GPU they come from GpuContext::memory_stats(). Use
--device all|cpu|gpu to select backends and --json path to write machine-readable
results.
A valid result submission should include criterion/ HTML or machine-readable
output, the exact commit hash, complete hardware metadata, and the command line.
Compare only equivalent precision, batch size, context length, transfer policy,
and synchronization boundaries.
Scope
OxideLM is an educational systems project, not a replacement for mature distributed-training frameworks. It prioritizes explicit mechanics, compact safe-Rust implementations, and measured selective fusion over mixed precision, quantization, distributed execution, and production model compatibility.
Contributing
Focused improvements are welcome. Good contribution areas include profiled kernel optimization, more tensor views, checkpoint compatibility, mixed precision, and reproducible benchmarks. Include correctness tests and before/after measurements for performance changes.
License
OxideLM is offered under either of the following licenses, at your option:
Unless explicitly stated otherwise, contributions intentionally submitted for inclusion in OxideLM are dual-licensed under the same terms.
Acknowledgments
OxideLM draws inspiration from the GPT and Llama decoder families, the original
Transformer and BPE literature, and the Rust, Rayon, and wgpu ecosystems. The
implementation remains independent and is intended to make the underlying
systems ideas easier to inspect, modify, and understand.