oxidizr 0.1.0-beta1

A Rust-based LLM training framework built on Candle
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

# GPU Training

GPU acceleration significantly speeds up training. Oxidizr uses CUDA for GPU support.

## Requirements

- NVIDIA GPU with CUDA support
- CUDA Toolkit 12.x installed
- `nvidia-smi` should work

## Building with CUDA

```bash
cargo build --release --features cuda
```

## Running on GPU

```bash
# GPU is default when CUDA feature is enabled
cargo run --release --features cuda -- -f models/nano.yaml

# Explicit GPU target
cargo run --release --features cuda -- -f models/nano.yaml --target-device gpu
```

## Performance

GPU training is significantly faster:

| Model | CPU | GPU (RTX 3090) |
|-------|-----|----------------|
| nano (~83M) | ~3 it/s | ~50+ it/s |

## Memory Management

### Check GPU Memory

```bash
nvidia-smi
```

### Adjust for Your GPU

```bash
# If out of memory, reduce batch size
cargo run --release --features cuda -- -f models/nano.yaml --batch-size 2

# Or reduce sequence length
cargo run --release --features cuda -- -f models/nano.yaml --seq-len 256

# Or use gradient accumulation (config file)
```

### Memory Estimation

Oxidizr estimates memory requirements before training starts. Watch for warnings about VRAM usage.

## Precision Options

Use lower precision for faster training and less memory:

```bash
# BF16 (recommended for modern GPUs - Ampere+)
cargo run --release --features cuda -- -f models/nano.yaml --dtype bf16

# FP16
cargo run --release --features cuda -- -f models/nano.yaml --dtype f16

# FP32 (default, most precise)
cargo run --release --features cuda -- -f models/nano.yaml --dtype f32
```

**BF16** is recommended for RTX 30xx/40xx and A100/H100.

## Selecting GPU

Use specific GPU by ID:

```bash
# Use GPU 0
CUDA_VISIBLE_DEVICES=0 cargo run --release --features cuda -- -f models/nano.yaml

# Use GPU 1
CUDA_VISIBLE_DEVICES=1 cargo run --release --features cuda -- -f models/nano.yaml
```

## Multi-GPU

See [Multi-GPU Training](multi-gpu.md) for training across multiple GPUs.

## Troubleshooting

**CUDA not found?**
- Ensure CUDA 12.x is installed
- Check `nvcc --version`
- Verify `nvidia-smi` works

**Out of memory?**
- Reduce `--batch-size`
- Reduce `--seq-len`
- Use `--dtype bf16` or `--dtype f16`
- Use gradient accumulation

**Wrong GPU used?**
- Set `CUDA_VISIBLE_DEVICES=X` before command

**Build errors?**
- CUDA 13.x not yet supported (use CUDA 12.x)
- Ensure CUDA toolkit matches your driver