Skip to main content

Crate rusting_brain

Crate rusting_brain 

Source
Expand description

A deep-learning library in Rust, from a two-layer XOR network up to a 300M-parameter transformer language model.

There are two model types, and they share the matrix, optimizer, and serialization code underneath:

§A language model in twenty lines

use rusting_brain::{Optimizer, TransformerLm};

let mut model = TransformerLm::builder()
    .vocab_size(32_000)
    .d_model(512)
    .n_layers(8)
    .heads(8, 2, 64)          // 8 query heads, 2 key/value heads (GQA)
    .d_ff(1408)
    .experts(8, 2)            // 8 experts, 2 active per token
    .moe_layers(2..8)         // layers 0-1 stay dense
    .max_seq_len(1024)
    .optimizer(Optimizer::adam(3e-4))
    .seed(42)
    .build()?;

println!("{}", model.parameter_counts());   // total vs. active

let batch: Vec<Vec<u32>> = vec![vec![1, 2, 3, 4], vec![5, 6, 7, 8]];
let loss = model.train_step(&batch)?;
println!("loss {:.4}", loss.lm_loss);

§Generating text

TransformerLm::generate prefills a per-layer KvCache with the prompt and then decodes one token at a time, so the n-th token costs one row of attention instead of n. Sampler decides which token that is.

let mut sampler = Sampler::temperature(0.8, Some(42)).top_k(40).top_p(0.95);
let generated = model.generate(&prompt, 50, &mut sampler)?;

TransformerLm::forward_cached is the cache underneath, for driving decoding by hand.

§Running on a GPU

CUDA is an optional feature and is off by default; the crate builds and the tests pass without a driver, a toolkit, or a device.

cargo add rusting_brain --features cuda

[TransformerLm::to_cuda] moves the parameters onto a device and every later train_step, forward_batch, and backward runs there. The device path is fail-closed: if the driver, cuBLAS, a kernel, an allocation, or a numerical check fails, the call returns NetworkError rather than silently falling back to the CPU.

model.set_mixed_precision(true);      // BF16 GEMMs, FP32 master weights
model.to_cuda(0, 9_000)?;             // device 0, 9000 MiB budget

Dense Networks use a separate device path, TrainingBackend::Cuda, or metal_training on Apple Silicon.

  • tutorials/ — an eighteen-chapter course, from what a neural network is to training a language model end to end.
  • docs/baseline.md — measured throughput per architecture on an RTX 3060.
  • IMPORT_MODELS.md — running ONNX models exported from TensorFlow or PyTorch.

Re-exports§

pub use accelerator::AcceleratorDoctorReport;
pub use accelerator::AcceleratorStats;
pub use accelerator::TrainingSession;
pub use accelerator::accelerator_doctor;
pub use accelerator::estimate_tensor_memory_mib;
pub use activations::Activation;
pub use adaln::AdaLayerNorm;
pub use adaln::AdaLnCache;
pub use adaln::Modulation;
pub use adaln::ModulationCache;
pub use adaln::TimestepCache;
pub use adaln::TimestepEmbedding;
pub use adaln::gate_residual;
pub use adaln::gate_residual_backward;
pub use attention::CrossAttentionCache;
pub use attention::KvCache;
pub use attention::MultiHeadAttention;
pub use batch::Layout;
pub use batch::TokenBatch;
pub use causal_lm_loss::CausalLmLoss;
pub use causal_lm_loss::TotalLoss;
pub use causal_lm_loss::causal_lm_loss;
pub use causal_lm_loss::causal_lm_loss_batch;
pub use checkpoint::load as load_checkpoint;
pub use checkpoint::save as save_checkpoint;
pub use clip::ClipTextConfig;
pub use clip::ClipTextEncoder;
pub use conv::Conv2d;
pub use conv::ConvCache;
pub use conv::ConvGeometry;
pub use conv::FeatureMap;
pub use conv::GroupNorm;
pub use conv::GroupNormCache;
pub use conv::ImageBatch;
pub use conv::TrainableConv2d;
pub use conv::TrainableGroupNorm;
pub use conv::pixel_shuffle;
pub use conv::pixel_unshuffle;
pub use conv::upsample_nearest;
pub use dataset::Augment;
pub use dataset::BatchCursor;
pub use dataset::BatchSource;
pub use dataset::Dataset;
pub use dataset::DatasetBatch;
pub use dataset::DatasetStream;
pub use dataset::JsonlStream;
pub use dataset::Standardizer;
pub use diffusion::Denoiser;
pub use diffusion::Inpaint;
pub use diffusion::SamplingConfig;
pub use diffusion::Scheduler;
pub use diffusion::Solver;
pub use diffusion::noise;
pub use diffusion::sample;
pub use diffusion::sample_consistency;
pub use diffusion::sample_from;
pub use diffusion::sample_unipc;
pub use diffusion::sample_with_sigmas;
pub use ema::Ema;
pub use embedding::Embedding;
pub use ffn::GeluMlp;
pub use ffn::SwiGlu;
pub use flow_transformer::FlowConfig;
pub use flow_transformer::FlowDenoiser;
pub use flow_transformer::FlowTransformer;
pub use flow_transformer::flow_match_target;
pub use flow_transformer::sample_timesteps;
pub use gltf::Glb;
pub use losses::Loss;
pub use masked_lm::MaskedBatch;
pub use masked_lm::masked_lm_loss;
pub use matrix::Matrix;
pub use mesh::Bvh;
pub use mesh::Mesh;
pub use mesh::QuerySampling;
pub use mesh::Transform;
pub use mesh::marching_tetrahedra;
pub use mesh::marching_tetrahedra_sparse;
pub use mmdit::Conditioning;
pub use mmdit::Dit;
pub use mmdit::DitConfig;
pub use moe::Expert;
pub use moe::MoeConfig;
pub use moe::MoeLayer;
pub use moe::Router;
pub use network::CudaTrainingCheckpoint;
pub use network::Dense;
pub use network::DenseLayer;
pub use network::Network;
pub use network::NetworkBuilder;
pub use network::NetworkError;
pub use network::TrainConfig;
pub use network::TrainingBackend;
pub use network::TrainingHistory;
pub use norm::RmsNorm;
pub use optimizers::Optimizer;
pub use optimizers::Schedule;
pub use optimizers::SpikeGuard;
pub use param::CudaDevice;
pub use param::Linear;
pub use param::Lora;
pub use param::Param;
pub use pipeline::DynamicShift;
pub use pipeline::ImageDenoiser;
pub use pipeline::ImagePipeline;
pub use pipeline::PipelineConfig;
pub use pipeline::PooledEncoder;
pub use pipeline::PromptEncoder;
pub use rng::RunRng;
pub use rope::Rope;
pub use safetensors::Dtype;
pub use safetensors::SafeTensors;
pub use safetensors::ShardedSafeTensors;
pub use safetensors::TensorInfo;
pub use sampling::Sampler;
pub use shape_vae::ShapeVae;
pub use shape_vae::ShapeVaeConfig;
pub use shape_vae::fourier_features;
pub use shards::Batch;
pub use shards::BatchConfig;
pub use shards::BatchStream;
pub use shards::Corpus;
pub use shards::Example;
pub use shards::Shards;
pub use t5::T5Config;
pub use t5::T5Encoder;
pub use text_encoder::TextEncoder;
pub use text_encoder::TextEncoderConfig;
pub use token_file::TokenFile;
pub use token_file::TokenMix;
pub use token_file::TokenStream;
pub use tokenizer::Bpe;
pub use tokenizer::Unigram;
pub use transformer::BatchDecoder;
pub use transformer::Decoder;
pub use transformer::LoraConfig;
pub use transformer::ParameterCounts;
pub use transformer::Precision;
pub use transformer::TransformerBuilder;
pub use transformer::TransformerConfig;
pub use transformer::TransformerLm;
pub use transformer_block::FeedForward;
pub use transformer_block::TransformerBlock;
pub use unet::Control;
pub use unet::ControlNet;
pub use unet::Unet;
pub use unet::UnetConfig;
pub use vae::VaeConfig;
pub use vae::VaeDecoder;
pub use vae::VaeEncoder;
pub use vae::to_rgb8;
pub use video_pipeline::UniPc;
pub use video_pipeline::Video;
pub use video_pipeline::VideoConfig;
pub use video_pipeline::VideoPipeline;
pub use video_pipeline::VideoRequest;
pub use vision::VisionTransformer;
pub use vision::VitConfig;
pub use vit_encoder::CLIP_MEAN;
pub use vit_encoder::CLIP_STD;
pub use vit_encoder::VitEncoder;
pub use vit_encoder::VitEncoderConfig;
pub use vit_encoder::composite_rgba;
pub use wan_dit::WanBlock;
pub use wan_dit::WanDit;
pub use wan_dit::WanDitConfig;

Modules§

accelerator
Backend-agnostic accelerator training.
activations
Element-wise activations for dense layers.
adaln
Adaptive layer normalization: a block whose normalization is steered by a conditioning vector rather than by weights alone.
attention
Grouped-query self-attention with rotary positions and a causal mask.
batch
Packing several sequences into one activation matrix.
causal_lm_loss
Next-token cross-entropy, plus the auxiliary losses MoE layers report.
checkpoint
Saving and restoring a training run, across as many models as it has.
clip
The CLIP text tower, which is the prompt encoder every Stable Diffusion model and FLUX.1 read their pooled vector from.
conv
Convolutions and the layers that surround them in an image decoder.
dataset
In-memory tabular data: rows of inputs paired with rows of targets.
dcae
The DC-AE decoder (AutoencoderDC, dc-ae-f32c32-sana-1.1): 32 latent channels at 1/32 scale back to RGB.
detection
Object detection data in the YOLO format: a reader for images/<split> and labels/<split> directories, and the resizing, letterboxing, mosaic and horizontal flip that ultralytics applies, with the same geometry so that boxes land where its pipeline puts them.
diffusion
Running a diffusion image model: the sampling loop and the noise schedules.
dit_lora
Low-rank adapters on the DiT’s block projections, and the backward pass that trains them.
ema
An exponential moving average of the weights, kept beside the model.
embedding
Token embedding, optionally shared with the output projection.
ffn
Position-wise feed-forward networks.
flow_transformer
The flow-matching transformer that turns an image into a shape latent.
gguf
Writing a TransformerLm as a GGUF file, the format llama.cpp and Ollama run.
gltf
Reading glTF binary files, which is what a 3D asset dataset is made of.
hf
Reading and writing a TransformerLm in the layout Hugging Face publishes LLaMA-family models in: a config.json beside a model.safetensors.
interoceptive
An interoceptive trading agent: a Q-network whose market encoder is steered by a small model of the account’s own “body” rather than fed it as input.
losses
Loss functions for dense networks.
masked_lm
Masked language modelling: the encoder-only training objective.
matrix
Row-major f32 matrices and the matmuls the CPU path is built on.
mesh
Triangle meshes, and turning them into the signed distances a shape model trains on.
mmdit
The transformer that denoises a latent: MMDiT, as FLUX and Stable Diffusion 3 build it.
moe
Sparse mixture-of-experts feed-forward layers.
network
Dense feed-forward networks: layers, the builder, and the training loop.
norm
Normalization layers. RMSNorm is what the decoder blocks use.
onnx
Inference for ONNX models trained in another framework.
optimizers
Parameter update rules.
param
Trainable tensors for the transformer stack.
pipeline
Prompt in, image out: the four parts of a text-to-image model wired together.
quantized
Weights stored as one byte per value, or half a byte, for inference.
recurrent
A GRU cell with backward through time.
rng
A random number generator a training run can checkpoint.
rope
Rotary position embeddings.
safetensors
Reading and writing the weight format every published model ships in.
sampling
Turning a row of logits into the next token id.
sana
SANA-Sprint 0.6B: a one-to-four step text-to-image model.
serialization
JSON save/load for both model types.
shape_vae
A variational autoencoder over shapes, from surface points to a signed distance field.
shards
Reading the preprocessed corpus the training loop is fed from.
t5
The T5 encoder, which is the prompt encoder FLUX.1 and Stable Diffusion 3 read their per-token states from.
text_encoder
The text encoder a diffusion model conditions on.
token_file
Training on a corpus that does not fit in memory.
tokenizer
Byte-level byte-pair encoding, read from a tokenizer.json.
transformer
Decoder-only transformer assembly: configuration, builder, and the model.
transformer_block
One decoder block: pre-norm attention, then a pre-norm feed-forward.
unet
The UNet denoiser, which is what Stable Diffusion 1.x, 2.x and XL are.
unet_lora
Low-rank adapters on a UNet’s attention projections, and the backward pass that trains them.
vae
The decoder that turns a diffusion model’s latent into pixels.
video_pipeline
Wan 2.2 TI2V-5B, prompt (and optionally a first frame) in, video out.
vision
Vision transformer: an image as a sequence of patches.
vit_encoder
The CLIP image tower, frozen, as a source of conditioning tokens.
wan_dit
Wan 2.2’s video transformer, WanTransformer3DModel in diffusers.
wan_vae
Wan 2.2’s video autoencoder, AutoencoderKLWan in diffusers.
webdataset
WebDataset shards: tar files whose members are grouped into samples by name, as in 000017.jpg, 000017.txt, 000017.json. Everything up to the first dot of the file name (directory included) is the sample’s key, and the rest is the extension under which that member is stored. A sample’s members must sit next to each other in the shard, which is how webdataset writes them and what tar of a sorted directory gives.
yolo
YOLO11, ultralytics’ yolo11.yaml, as a graph over the layers in crate::conv: trainable on the host and on a device, with every tensor named as in an ultralytics state dict so trained weights load directly.
zimage
Z-Image-Turbo’s denoiser (Tongyi-MAI, Apache-2.0): diffusers’ ZImageTransformer2DModel.

Constants§

BUILD
The library version and the commit it was built from, such as 3.0.0 (b46a6b4). Print it at startup: a binary built from an older checkout then shows a hash that does not match git rev-parse --short HEAD. The hash is unknown outside a git checkout.