# RustingBrain
A deep-learning library in Rust, from a two-layer XOR network up to a
300M-parameter transformer language model trained on one desktop GPU.
No Python in the loop, no C++ build step, no framework runtime. The CUDA
kernels are compiled at startup by NVRTC and the rest is Rust.
## Speed
Training tokens per second for the same 101.7M-parameter transformer, mixed
precision on every side, `torch.compile` for PyTorch and XLA for TensorFlow, on
one idle RTX 3060 12 GB:
| 1 × 512 | 13071 | 15560 | 9607 |
| 1 × 1024 | **16352** | 15652 | 12235 |
| 4 × 512 | 22045 | 22412 | 17993 |
| 4 × 1024 | **21787** | 20063 | 20544 |
| 8 × 512 | 22926 | 23906 | 21366 |
Single GPU, one architecture family. Other shapes are in
[docs/baseline.md](docs/baseline.md).
## Features
- **Language models**: decoder-only transformers with grouped-query attention,
rotary positions, RMSNorm, SwiGLU, flash attention and mixture-of-experts
layers.
- **Training**: mixed precision, gradient accumulation, clipping, schedules,
LoRA, resumable checkpoints, corpora larger than memory.
- **Inference**: KV-cached and streaming generation, samplers, int8 weights.
- **Images and video**: FLUX, FLUX.2, Stable Diffusion 1.x to 3 and Wan 2.2,
loaded from the published safetensors directory.
- **Dense networks**: regression and classification with CSV, `.npy`, IDX and
image-folder loaders.
- **Backends**: CPU, CUDA, and Metal for dense networks.
- **Interop**: ONNX import and export, Hugging Face tokenizers and checkpoints.
## Install
```bash
cargo add rusting_brain # CPU only
cargo add rusting_brain --features cuda # NVIDIA GPU
```
The crate builds and tests with no GPU present. Driver and toolkit setup is in
[INSTALL_CUDA.md](INSTALL_CUDA.md). Other features: `onnx`, `images`, `metal`.
## A language model
```rust
use rusting_brain::{Optimizer, TransformerLm};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut model = TransformerLm::builder()
.vocab_size(32_000)
.d_model(512)
.n_layers(8)
.heads(8, 2, 64) // 8 query heads, 2 key/value heads
.d_ff(1408)
.experts(8, 2) // 8 experts, 2 active per token
.moe_layers(2..8) // layers 0 and 1 stay dense
.max_seq_len(1024)
.optimizer(Optimizer::adam(3e-4))
.seed(42)
.build()?;
model.set_mixed_precision(true);
model.to_cuda(0, 9_000)?; // device 0, 9000 MiB budget; skip for CPU
let batch: Vec<Vec<u32>> = vec![vec![1, 2, 3, 4], vec![5, 6, 7, 8]];
let loss = model.train_step(&batch)?;
println!("loss {:.4}", loss.lm_loss);
Ok(())
}
```
## A dense network
```rust
let mut model = Network::builder()
.input_size(2)
.dense(8, Activation::Tanh)
.dense(1, Activation::Sigmoid)
.loss(Loss::BinaryCrossEntropy)
.optimizer(Optimizer::adam(0.05))
.build();
model.fit(&data, TrainConfig { epochs: 2_000, batch_size: 4, shuffle: true, seed: Some(42) })?;
println!("{:?}", model.predict(&[1.0, 0.0])?);
```
## Text to image
```rust
let mut pipeline = ImagePipeline::load("models/FLUX.2-klein-4B")?;
## Scope
One machine, one GPU. Encoder-decoder models, multi-GPU and distributed
training are not implemented and are not planned.
## License
RustingBrain License 1.1: use, modify and distribute it, and sell projects
built with it, but do not sell RustingBrain itself. See
[LICENSE.md](LICENSE.md).