oxidizr 0.1.0-beta1

A Rust-based LLM training framework built on Candle
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

# CPU Training

Oxidizr supports full training on CPU without any GPU. This is great for learning, prototyping, and systems without GPU access.

## Building for CPU

```bash
# Standard build (no special flags needed)
cargo build --release
```

## Running on CPU

```bash
# Explicit CPU target
cargo run --release -- -f models/nano.yaml --target-device cpu

# Or just run without CUDA feature (defaults to CPU)
cargo run --release -- -f models/nano.yaml
```

## Performance Expectations

CPU training is slower than GPU but fully functional:

| Model | CPU Speed | GPU Speed |
|-------|-----------|-----------|
| nano (~83M params) | ~3 it/s | ~50+ it/s |

Actual speed depends on:
- CPU cores and speed
- Batch size and sequence length
- Model architecture

## Optimizing CPU Training

### Reduce Memory Pressure

```bash
# Smaller batch size
cargo run --release -- -f models/nano.yaml --batch-size 1 --target-device cpu

# Shorter sequences
cargo run --release -- -f models/nano.yaml --seq-len 128 --target-device cpu
```

### Use Gradient Accumulation

Simulate larger batches without more memory:

```yaml
trainer:
  batch_size: 1
  gradient_accumulation: 4  # Effective batch = 4
```

### Enable Prefetching

Overlap data loading with computation:

```bash
cargo run --release -- -f models/nano.yaml --prefetch 2 --target-device cpu
```

## When to Use CPU

- **Learning** - Understand the training process
- **Prototyping** - Test config changes quickly
- **No GPU available** - Cloud instances, laptops
- **Debugging** - Easier to debug without CUDA
- **Small models** - Nano-scale experiments

## Comparing CPU vs GPU

```bash
# Test on CPU
cargo run --release -- -f models/nano.yaml --target-device cpu --max-steps 100 --headless

# Test on GPU (requires CUDA build)
cargo build --release --features cuda
cargo run --release --features cuda -- -f models/nano.yaml --target-device gpu --max-steps 100 --headless
```

## Troubleshooting

**Training seems slow?**
- This is expected. CPU is 10-20x slower than GPU.
- Use `--headless` to see actual iteration speed.

**Out of memory?**
- Reduce batch size: `--batch-size 1`
- Reduce sequence length: `--seq-len 128`

**Want to use GPU?**
- See [GPU Training](gpu.md)