aprender 0.31.2

Next-generation ML framework in pure Rust — `cargo install aprender` for the `apr` CLI
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

<!-- PCU: examples-dpo-preference | contract: contracts/apr-page-examples-dpo-preference-v1.yaml -->
<!-- Example: cargo run -p aprender-core --example dpo_preference -->
<!-- Status: enforced -->

# Case Study: Direct Preference Optimization (DPO)

**Ticket**: GH-449 | **Contract**: `finetune.md §dpo`

## Overview

DPO (Rafailov et al. 2023) aligns language models using preference pairs
without training a separate reward model. The policy is optimized directly
to maximize the margin between chosen and rejected responses.

**Loss**: `L = -log σ(β · (log π(y_w|x)/π_ref(y_w|x) - log π(y_l|x)/π_ref(y_l|x)))`

## API

```rust
use aprender::online::dpo::{DpoConfig, DpoLoss, DpoMetrics, PreferencePair};

let loss = DpoLoss::new(DpoConfig {
    beta: 0.1,
    label_smoothing: 0.0,
    use_reference: true,
    ..Default::default()
});

let pair = PreferencePair {
    chosen_logprob: -1.0,
    rejected_logprob: -5.0,
    ref_chosen_logprob: -2.0,
    ref_rejected_logprob: -3.0,
};

let l = loss.compute(&pair);
let (grad_c, grad_r) = loss.gradient(&pair);  // Zero-sum gradients
let metrics = DpoMetrics::from_batch(&loss, &pairs);
```

## Key Properties

- **Zero-sum gradients**: grad_chosen + grad_rejected = 0
- **Non-negative loss**: DPO loss is always >= 0
- **SimPO variant**: Set `use_reference: false` for reference-free training
- **Label smoothing**: Reduces overfitting to noisy preference annotations

## Falsification Tests

| Test | Property |
|------|----------|
| FALSIFY-DPO-001 | Loss is always non-negative |
| FALSIFY-DPO-002 | Gradients are zero-sum |
| FALSIFY-DPO-003 | Correct reward ordering when policy is correct |

## Run the Example

```bash
cargo run --example dpo_preference
```