Expand description
A deep-learning library in Rust, from a two-layer XOR network up to a 300M-parameter transformer language model.
There are two model types, and they share the matrix, optimizer, and serialization code underneath:
Network— a dense feed-forward network for tabular regression and classification. Built withNetwork::builder, trained withNetwork::fit.TransformerLm— a decoder-only language model with grouped-query attention, rotary positions, RMSNorm, SwiGLU feed-forwards, and optional sparse mixture-of-experts layers. Built withTransformerLm::builder, trained one batch at a time withTransformerLm::train_step.
§A language model in twenty lines
use rusting_brain::{Optimizer, TransformerLm};
let mut model = TransformerLm::builder()
.vocab_size(32_000)
.d_model(512)
.n_layers(8)
.heads(8, 2, 64) // 8 query heads, 2 key/value heads (GQA)
.d_ff(1408)
.experts(8, 2) // 8 experts, 2 active per token
.moe_layers(2..8) // layers 0-1 stay dense
.max_seq_len(1024)
.optimizer(Optimizer::adam(3e-4))
.seed(42)
.build()?;
println!("{}", model.parameter_counts()); // total vs. active
let batch: Vec<Vec<u32>> = vec![vec![1, 2, 3, 4], vec![5, 6, 7, 8]];
let loss = model.train_step(&batch)?;
println!("loss {:.4}", loss.lm_loss);§Generating text
TransformerLm::generate prefills a per-layer KvCache with the prompt
and then decodes one token at a time, so the n-th token costs one row of
attention instead of n. Sampler decides which token that is.
let mut sampler = Sampler::temperature(0.8, Some(42)).top_k(40).top_p(0.95);
let generated = model.generate(&prompt, 50, &mut sampler)?;TransformerLm::forward_cached is the cache underneath, for driving
decoding by hand.
§Running on a GPU
CUDA is an optional feature and is off by default; the crate builds and the tests pass without a driver, a toolkit, or a device.
cargo add rusting_brain --features cuda[TransformerLm::to_cuda] moves the parameters onto a device and every
later train_step, forward_batch, and backward runs there. The device
path is fail-closed: if the driver, cuBLAS, a kernel, an allocation, or a
numerical check fails, the call returns NetworkError rather than
silently falling back to the CPU.
model.set_mixed_precision(true); // BF16 GEMMs, FP32 master weights
model.to_cuda(0, 9_000)?; // device 0, 9000 MiB budgetDense Networks use a separate device path,
TrainingBackend::Cuda, or metal_training on Apple Silicon.
§Where to read next
tutorials/— an eighteen-chapter course, from what a neural network is to training a language model end to end.docs/baseline.md— measured throughput per architecture on an RTX 3060.IMPORT_MODELS.md— running ONNX models exported from TensorFlow or PyTorch.
Re-exports§
pub use accelerator::AcceleratorDoctorReport;pub use accelerator::AcceleratorStats;pub use accelerator::TrainingSession;pub use accelerator::accelerator_doctor;pub use accelerator::estimate_tensor_memory_mib;pub use activations::Activation;pub use adaln::AdaLayerNorm;pub use adaln::AdaLnCache;pub use adaln::Modulation;pub use adaln::ModulationCache;pub use adaln::TimestepCache;pub use adaln::TimestepEmbedding;pub use adaln::gate_residual;pub use adaln::gate_residual_backward;pub use attention::CrossAttentionCache;pub use attention::KvCache;pub use attention::MultiHeadAttention;pub use batch::Layout;pub use batch::TokenBatch;pub use causal_lm_loss::CausalLmLoss;pub use causal_lm_loss::TotalLoss;pub use causal_lm_loss::causal_lm_loss;pub use causal_lm_loss::causal_lm_loss_batch;pub use checkpoint::load as load_checkpoint;pub use checkpoint::save as save_checkpoint;pub use clip::ClipTextConfig;pub use clip::ClipTextEncoder;pub use conv::Conv2d;pub use conv::ConvCache;pub use conv::ConvGeometry;pub use conv::FeatureMap;pub use conv::GroupNorm;pub use conv::GroupNormCache;pub use conv::ImageBatch;pub use conv::TrainableConv2d;pub use conv::TrainableGroupNorm;pub use conv::pixel_shuffle;pub use conv::pixel_unshuffle;pub use conv::upsample_nearest;pub use dataset::Augment;pub use dataset::BatchCursor;pub use dataset::BatchSource;pub use dataset::Dataset;pub use dataset::DatasetBatch;pub use dataset::DatasetStream;pub use dataset::JsonlStream;pub use dataset::Standardizer;pub use diffusion::Denoiser;pub use diffusion::Inpaint;pub use diffusion::SamplingConfig;pub use diffusion::Scheduler;pub use diffusion::Solver;pub use diffusion::noise;pub use diffusion::sample;pub use diffusion::sample_consistency;pub use diffusion::sample_from;pub use diffusion::sample_unipc;pub use diffusion::sample_with_sigmas;pub use ema::Ema;pub use embedding::Embedding;pub use ffn::GeluMlp;pub use ffn::SwiGlu;pub use flow_transformer::FlowConfig;pub use flow_transformer::FlowDenoiser;pub use flow_transformer::FlowTransformer;pub use flow_transformer::flow_match_target;pub use flow_transformer::sample_timesteps;pub use gltf::Glb;pub use losses::Loss;pub use masked_lm::MaskedBatch;pub use masked_lm::masked_lm_loss;pub use matrix::Matrix;pub use mesh::Bvh;pub use mesh::Mesh;pub use mesh::QuerySampling;pub use mesh::Transform;pub use mesh::marching_tetrahedra;pub use mesh::marching_tetrahedra_sparse;pub use mmdit::Conditioning;pub use mmdit::Dit;pub use mmdit::DitConfig;pub use moe::Expert;pub use moe::MoeConfig;pub use moe::MoeLayer;pub use moe::Router;pub use network::CudaTrainingCheckpoint;pub use network::Dense;pub use network::DenseLayer;pub use network::Network;pub use network::NetworkBuilder;pub use network::NetworkError;pub use network::TrainConfig;pub use network::TrainingBackend;pub use network::TrainingHistory;pub use norm::RmsNorm;pub use optimizers::Optimizer;pub use optimizers::Schedule;pub use optimizers::SpikeGuard;pub use param::CudaDevice;pub use param::Linear;pub use param::Lora;pub use param::Param;pub use pipeline::DynamicShift;pub use pipeline::ImageDenoiser;pub use pipeline::ImagePipeline;pub use pipeline::PipelineConfig;pub use pipeline::PooledEncoder;pub use pipeline::PromptEncoder;pub use rng::RunRng;pub use rope::Rope;pub use safetensors::Dtype;pub use safetensors::SafeTensors;pub use safetensors::ShardedSafeTensors;pub use safetensors::TensorInfo;pub use sampling::Sampler;pub use shape_vae::ShapeVae;pub use shape_vae::ShapeVaeConfig;pub use shape_vae::fourier_features;pub use shards::Batch;pub use shards::BatchConfig;pub use shards::BatchStream;pub use shards::Corpus;pub use shards::Example;pub use shards::Shards;pub use t5::T5Config;pub use t5::T5Encoder;pub use text_encoder::TextEncoder;pub use text_encoder::TextEncoderConfig;pub use token_file::TokenFile;pub use token_file::TokenMix;pub use token_file::TokenStream;pub use tokenizer::Bpe;pub use tokenizer::Unigram;pub use transformer::BatchDecoder;pub use transformer::Decoder;pub use transformer::LoraConfig;pub use transformer::ParameterCounts;pub use transformer::Precision;pub use transformer::TransformerBuilder;pub use transformer::TransformerConfig;pub use transformer::TransformerLm;pub use transformer_block::FeedForward;pub use transformer_block::TransformerBlock;pub use unet::Control;pub use unet::ControlNet;pub use unet::Unet;pub use unet::UnetConfig;pub use vae::VaeConfig;pub use vae::VaeDecoder;pub use vae::VaeEncoder;pub use vae::to_rgb8;pub use video_pipeline::UniPc;pub use video_pipeline::Video;pub use video_pipeline::VideoConfig;pub use video_pipeline::VideoPipeline;pub use video_pipeline::VideoRequest;pub use vision::VisionTransformer;pub use vision::VitConfig;pub use vit_encoder::CLIP_MEAN;pub use vit_encoder::CLIP_STD;pub use vit_encoder::VitEncoder;pub use vit_encoder::VitEncoderConfig;pub use vit_encoder::composite_rgba;pub use wan_dit::WanBlock;pub use wan_dit::WanDit;pub use wan_dit::WanDitConfig;
Modules§
- accelerator
- Backend-agnostic accelerator training.
- activations
- Element-wise activations for dense layers.
- adaln
- Adaptive layer normalization: a block whose normalization is steered by a conditioning vector rather than by weights alone.
- attention
- Grouped-query self-attention with rotary positions and a causal mask.
- batch
- Packing several sequences into one activation matrix.
- causal_
lm_ loss - Next-token cross-entropy, plus the auxiliary losses MoE layers report.
- checkpoint
- Saving and restoring a training run, across as many models as it has.
- clip
- The CLIP text tower, which is the prompt encoder every Stable Diffusion model and FLUX.1 read their pooled vector from.
- conv
- Convolutions and the layers that surround them in an image decoder.
- dataset
- In-memory tabular data: rows of inputs paired with rows of targets.
- dcae
- The DC-AE decoder (
AutoencoderDC,dc-ae-f32c32-sana-1.1): 32 latent channels at 1/32 scale back to RGB. - detection
- Object detection data in the YOLO format: a reader for
images/<split>andlabels/<split>directories, and the resizing, letterboxing, mosaic and horizontal flip that ultralytics applies, with the same geometry so that boxes land where its pipeline puts them. - diffusion
- Running a diffusion image model: the sampling loop and the noise schedules.
- dit_
lora - Low-rank adapters on the DiT’s block projections, and the backward pass that trains them.
- ema
- An exponential moving average of the weights, kept beside the model.
- embedding
- Token embedding, optionally shared with the output projection.
- ffn
- Position-wise feed-forward networks.
- flow_
transformer - The flow-matching transformer that turns an image into a shape latent.
- gguf
- Writing a
TransformerLmas a GGUF file, the format llama.cpp and Ollama run. - gltf
- Reading glTF binary files, which is what a 3D asset dataset is made of.
- hf
- Reading and writing a
TransformerLmin the layout Hugging Face publishes LLaMA-family models in: aconfig.jsonbeside amodel.safetensors. - interoceptive
- An interoceptive trading agent: a Q-network whose market encoder is steered by a small model of the account’s own “body” rather than fed it as input.
- losses
- Loss functions for dense networks.
- masked_
lm - Masked language modelling: the encoder-only training objective.
- matrix
- Row-major
f32matrices and the matmuls the CPU path is built on. - mesh
- Triangle meshes, and turning them into the signed distances a shape model trains on.
- mmdit
- The transformer that denoises a latent: MMDiT, as FLUX and Stable Diffusion 3 build it.
- moe
- Sparse mixture-of-experts feed-forward layers.
- network
- Dense feed-forward networks: layers, the builder, and the training loop.
- norm
- Normalization layers. RMSNorm is what the decoder blocks use.
- onnx
- Inference for ONNX models trained in another framework.
- optimizers
- Parameter update rules.
- param
- Trainable tensors for the transformer stack.
- pipeline
- Prompt in, image out: the four parts of a text-to-image model wired together.
- quantized
- Weights stored as one byte per value, or half a byte, for inference.
- recurrent
- A GRU cell with backward through time.
- rng
- A random number generator a training run can checkpoint.
- rope
- Rotary position embeddings.
- safetensors
- Reading and writing the weight format every published model ships in.
- sampling
- Turning a row of logits into the next token id.
- sana
- SANA-Sprint 0.6B: a one-to-four step text-to-image model.
- serialization
- JSON save/load for both model types.
- shape_
vae - A variational autoencoder over shapes, from surface points to a signed distance field.
- shards
- Reading the preprocessed corpus the training loop is fed from.
- t5
- The T5 encoder, which is the prompt encoder FLUX.1 and Stable Diffusion 3 read their per-token states from.
- text_
encoder - The text encoder a diffusion model conditions on.
- token_
file - Training on a corpus that does not fit in memory.
- tokenizer
- Byte-level byte-pair encoding, read from a
tokenizer.json. - transformer
- Decoder-only transformer assembly: configuration, builder, and the model.
- transformer_
block - One decoder block: pre-norm attention, then a pre-norm feed-forward.
- unet
- The UNet denoiser, which is what Stable Diffusion 1.x, 2.x and XL are.
- unet_
lora - Low-rank adapters on a UNet’s attention projections, and the backward pass that trains them.
- vae
- The decoder that turns a diffusion model’s latent into pixels.
- video_
pipeline - Wan 2.2 TI2V-5B, prompt (and optionally a first frame) in, video out.
- vision
- Vision transformer: an image as a sequence of patches.
- vit_
encoder - The CLIP image tower, frozen, as a source of conditioning tokens.
- wan_dit
- Wan 2.2’s video transformer,
WanTransformer3DModelindiffusers. - wan_vae
- Wan 2.2’s video autoencoder,
AutoencoderKLWanindiffusers. - webdataset
- WebDataset shards: tar files whose members are grouped into samples by
name, as in
000017.jpg,000017.txt,000017.json. Everything up to the first dot of the file name (directory included) is the sample’s key, and the rest is the extension under which that member is stored. A sample’s members must sit next to each other in the shard, which is howwebdatasetwrites them and whattarof a sorted directory gives. - yolo
- YOLO11, ultralytics’
yolo11.yaml, as a graph over the layers incrate::conv: trainable on the host and on a device, with every tensor named as in an ultralytics state dict so trained weights load directly. - zimage
- Z-Image-Turbo’s denoiser (Tongyi-MAI, Apache-2.0): diffusers’
ZImageTransformer2DModel.
Constants§
- BUILD
- The library version and the commit it was built from, such as
3.0.0 (b46a6b4). Print it at startup: a binary built from an older checkout then shows a hash that does not matchgit rev-parse --short HEAD. The hash isunknownoutside a git checkout.