1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
//! Parameter-group optimizers: several optimizers, and several learning
//! rates, over one model.
//!
//! A `burn` optimizer config builds one rule for every parameter of a module.
//! Training recipes often want more than that:
//!
//! - 2-D weight matrices train well with [`Muon`](burn::optim::Muon), while
//! embeddings, norms and biases are better served by
//! [`AdamW`](burn::optim::AdamW);
//! - groups want their own learning rates: the `NanoChat` recipe scales the
//! `lm_head` and embedding rates separately, by a `d_model` factor;
//! - groups want their own weight decay, betas or gradient clipping.
//!
//! `burn`'s [`ModuleOptimizer`](burn::optim::ModuleOptimizer) and
//! [`ModuleLrScheduler`](burn::optim::lr_scheduler::module_lr_scheduler::ModuleLrScheduler)
//! do the stepping: each mounts several optimizers (or schedulers) over
//! [`ParamGroup`](burn::module::ParamGroup)s, and a parameter takes the last
//! group that matches it. [`GroupOptimizerPlan`] builds both from parameter
//! groups selected by `ParamId`, and adds what they leave out: a check that
//! the groups partition the module, groups keyed so that they survive a
//! checkpoint, and learning rates derived from one global schedule.
//!
//! This module is behind the `train` feature, which is on by default.
//!
//! # Lifecycle
//!
//! 1. **Select `ParamId`s.** Choose each group's parameters by structure with
//! [`module::reflection`](crate::burner::module::reflection): build an
//! [`XmlModuleTree`](crate::burner::module::reflection::XmlModuleTree) over
//! the live module and select from it with `XPath`
//! ([`select_param_ids`](crate::burner::module::reflection::XmlModuleTree::select_param_ids)).
//! Any other way to name the ids works too: a visitor, or the fields
//! themselves.
//! 2. **Build [`OptimizerGroup`]s.** A group is a set of `ParamId`s, the
//! optimizer that steps them (any `burn`
//! [`Optimizer`](burn::optim::Optimizer), such as
//! `AdamWConfig::new().build()`), and optionally an [`LrSelector`] and a
//! gradient clipping. [`OptimizerGroup::frozen`] builds a group whose
//! parameters are never moved ([`FrozenOptimizer`]). Groups of different
//! optimizer types go in one `Vec`.
//! 3. **Plan with [`GroupOptimizerPlan::try_new`].** It takes the module and
//! the groups, checks that the groups partition the module's float
//! parameters (none in two groups, none in no group), and keys each group by
//! the module paths of its parameters.
//! [`try_new_with_policy`](GroupOptimizerPlan::try_new_with_policy) takes an
//! [`UnknownParamPolicy`]; `try_new` uses the default, `Panic`.
//! 4. **Build and train.** [`GroupOptimizerPlan::optimizer`] builds the
//! `ModuleOptimizer`, and [`GroupOptimizerPlan::lr_scheduler`] the
//! `ModuleLrScheduler` over a global schedule. Hand both to a
//! [`Learner`](burn::train::Learner) (`Learner::new(model, plan.optimizer(),
//! plan.lr_scheduler(schedule))`), or step them in a loop of your own.
//!
//! # Resuming from a checkpoint
//!
//! A plan keys its groups by module path, such as `body.weight`, not by the
//! `ParamId`s they were selected with. Paths are what survive a checkpoint:
//!
//! - A module record saves each parameter's id, and `Module::load_record` gives
//! the parameter that id back. So a restored model has the saved run's ids,
//! not the ones it was built with.
//! - An optimizer record saves each parameter's state keyed by that id, with
//! its module path, and `ModuleOptimizer::load_record` hands each state to
//! the group that matches the saved path.
//!
//! So a resumed run is built like a first run: build a fresh model, select
//! its groups, and plan them, with the same groups in the same order as the
//! run that saved the checkpoint. Its fresh ids select the same paths. Then
//! load the module record into the model and the optimizer record into
//! `plan.optimizer()`. A [`Learner`](burn::train::Learner) resuming from a
//! checkpoint does the loading.
//!
//! The scheduler's record is matched by position: build the resumed
//! scheduler from the same plan shape, over the same base schedule.
//!
//! ```rust
//! use bunsen::{
//! burner::optim::{
//! GroupOptimizerPlan,
//! OptimizerGroup,
//! },
//! errors::BunsenResult,
//! support::testing::cpu_device,
//! };
//! use burn::{
//! module::Module,
//! nn::{
//! Linear,
//! LinearConfig,
//! },
//! optim::{
//! AdamWConfig,
//! GradientsParams,
//! ModuleOptimizer,
//! SgdConfig,
//! },
//! store::ModuleRecord,
//! tensor::{
//! Device,
//! Tensor,
//! },
//! };
//!
//! /// Builds the model and its optimizer: the same code starts a run and
//! /// resumes one. SGD steps the weight, `AdamW` the bias.
//! fn build(device: &Device) -> BunsenResult<(Linear, ModuleOptimizer)> {
//! let net: Linear = LinearConfig::new(4, 2).init(device);
//! let plan = GroupOptimizerPlan::try_new(
//! &net,
//! vec![
//! OptimizerGroup::new([net.weight.id], SgdConfig::new().build()),
//! OptimizerGroup::new(
//! [net.bias.as_ref().unwrap().id],
//! AdamWConfig::new().build(),
//! ),
//! ],
//! )?;
//! Ok((net, plan.optimizer()))
//! }
//!
//! fn train_step(
//! net: Linear,
//! optim: &mut ModuleOptimizer,
//! device: &Device,
//! ) -> Linear {
//! let loss = net.forward(Tensor::<2>::ones([3, 4], device)).sum();
//! let grads = GradientsParams::from_grads(loss.backward(), &net);
//! optim.step(1e-2, net, grads)
//! }
//!
//! let device = cpu_device().autodiff();
//!
//! // The first run trains, and saves a checkpoint.
//! let (net, mut optim) = build(&device)?;
//! let net = train_step(net, &mut optim, &device);
//! let model_bytes = net.into_record().into_bytes()?;
//! let optim_bytes = optim.into_bytes()?;
//!
//! // The resumed run builds a fresh model, with fresh `ParamId`s, and a plan
//! // over it, then loads both records. The `AdamW` state lands on the bias.
//! let (net, optim) = build(&device)?;
//! let net = net.load_record(ModuleRecord::from_bytes(model_bytes)?);
//! let mut optim = optim.from_bytes(optim_bytes)?;
//!
//! let before = net.weight.val().into_data();
//! let net = train_step(net, &mut optim, &device);
//! assert_ne!(net.weight.val().into_data(), before);
//! # Ok::<(), Box<dyn std::error::Error>>(())
//! ```
//!
//! # Behaviour to know
//!
//! - **Duplicates are rejected.** A `ParamId` in two groups makes `try_new`
//! return [`GroupOptimizerError::DuplicateParamId`].
//! - **Every float parameter needs a group.** A float parameter of the module
//! in no group makes `try_new` return
//! [`GroupOptimizerError::UnassignedParamIds`], listing each one with its
//! path, under the default policy (below). This includes a parameter with
//! `require_grad` off, which gets no gradient and so is not stepped whatever
//! its group. Put parameters you mean to keep fixed in a frozen group
//! ([`OptimizerGroup::frozen`]). Build a remnant group from the parameters no
//! other group claims.
//! - **The plan is keyed by paths.** A parameter is stepped by the group that
//! holds its module path. A model with the same structure as the planned one,
//! such as a freshly built or a restored one, is stepped the same way. A
//! parameter at a path the plan doesn't hold, such as a head under a new
//! field, goes to the fallback, which the [`UnknownParamPolicy`] picks. Plan
//! the groups after surgery.
//! - **The [`UnknownParamPolicy`] decides what a parameter in no group does.**
//! `Panic`, the default, fails: `try_new` returns `UnassignedParamIds`, and
//! `step` panics on the first such parameter, naming its rank and shape.
//! `Freeze` accepts the module and leaves such parameters unchanged.
//! - **Empty groups are not errors.** An `XPath` selection that matches nothing
//! gives an empty group, and the remnant group then claims the parameters it
//! meant to, with the remnant's settings. Assert that each group is
//! non-empty. A `ParamId` that is not a float parameter of the module is
//! ignored.
//! - **Only float parameters are stepped.** Int and bool parameters keep their
//! values, whatever group they are in, and need no group.
//! - **Gradient clipping is per group**
//! ([`OptimizerGroup::with_grad_clipping`]), and applies to each parameter's
//! gradient on its own, before its group's step: a norm clip bounds each
//! tensor's norm, not a global norm.
//!
//! # Learning rates
//!
//! [`GroupOptimizerPlan::lr_scheduler`] takes the *global* schedule, any
//! `burn` [`LrScheduler`](burn::optim::lr_scheduler::LrScheduler) (a constant
//! `LearningRate` is one), and each group maps its rate through its
//! [`LrSelector`]:
//!
//! - no selector, or [`GlobalLrSelector`]: the global rate;
//! - [`OptimizerGroup::with_fixed_lr`] ([`FixedLrSelector`]): a constant that
//! ignores the schedule;
//! - a closure, e.g. a per-group factor on the scheduled rate, as in the
//! example below. A `Send + Sync` closure from the global rate to the group's
//! rate is an `LrSelector`.
//!
//! A group with a selector gets a [`SelectedLr`] scheduler: the global
//! schedule, stepped in lockstep, mapped through the selector.
//!
//! # Example
//!
//! Muon for the matrices, `AdamW` at half the scheduled rate for the biases:
//!
//! ```rust
//! use bunsen::{
//! burner::optim::{
//! GroupOptimizerError,
//! GroupOptimizerPlan,
//! OptimizerGroup,
//! },
//! support::testing::cpu_device,
//! };
//! use burn::{
//! module::{
//! Module,
//! ParamId,
//! },
//! nn::{
//! Linear,
//! LinearConfig,
//! },
//! optim::{
//! AdamWConfig,
//! GradientsParams,
//! MuonConfig,
//! },
//! tensor::Tensor,
//! };
//!
//! #[derive(Module, Debug)]
//! struct Net {
//! body: Linear,
//! head: Linear,
//! }
//!
//! let device = cpu_device().autodiff();
//! let net: Net = Net {
//! body: LinearConfig::new(4, 4).init(&device),
//! head: LinearConfig::new(4, 2).init(&device),
//! };
//!
//! // 1. Select.
//! let matrices: Vec<ParamId> = vec![net.body.weight.id, net.head.weight.id];
//! let biases: Vec<ParamId> = [&net.body, &net.head]
//! .map(|linear| linear.bias.as_ref().unwrap().id)
//! .into();
//!
//! // 2. Groups.
//! let matrix_group = OptimizerGroup::new(matrices.clone(), MuonConfig::new().build());
//! let bias_group = OptimizerGroup::new(biases.clone(), AdamWConfig::new().build())
//! .with_lr_selector(|lr| lr * 0.5);
//!
//! // 3. Plan: the module, then the groups.
//! let plan = GroupOptimizerPlan::try_new(&net, vec![matrix_group, bias_group])?;
//!
//! // 4. Build, and step; or hand both to a `Learner`.
//! let mut optim = plan.optimizer();
//! let mut scheduler = plan.lr_scheduler(1e-2);
//! let before = net.body.weight.val().into_data();
//! let x = Tensor::<2>::ones([3, 4], &device);
//! let loss = net.head.forward(net.body.forward(x)).sum();
//! let grads = GradientsParams::from_grads(loss.backward(), &net);
//! let net = optim.step(scheduler.step(), net, grads);
//! assert_ne!(net.body.weight.val().into_data(), before);
//!
//! // A parameter claimed by two groups is rejected.
//! let twice = GroupOptimizerPlan::try_new(
//! &net,
//! vec![
//! OptimizerGroup::new(matrices.clone(), MuonConfig::new().build()),
//! OptimizerGroup::new(matrices.clone(), AdamWConfig::new().build()),
//! ],
//! );
//! assert!(matches!(
//! twice.unwrap_err().find::<GroupOptimizerError>(),
//! Some(GroupOptimizerError::DuplicateParamId { .. })
//! ));
//!
//! // So is a parameter claimed by none: here, the biases.
//! let partial = GroupOptimizerPlan::try_new(
//! &net,
//! vec![OptimizerGroup::new(matrices.clone(), MuonConfig::new().build())],
//! );
//! assert!(matches!(
//! partial.unwrap_err().find::<GroupOptimizerError>(),
//! Some(GroupOptimizerError::UnassignedParamIds { params }) if params.len() == 2
//! ));
//!
//! // To keep the biases fixed, freeze them in a group of their own.
//! let frozen = GroupOptimizerPlan::try_new(
//! &net,
//! vec![
//! OptimizerGroup::new(matrices, MuonConfig::new().build()),
//! OptimizerGroup::frozen(biases),
//! ],
//! )?;
//! let bias = net.body.bias.as_ref().unwrap().val().into_data();
//! let x = Tensor::<2>::ones([3, 4], &device);
//! let loss = net.head.forward(net.body.forward(x)).sum();
//! let grads = GradientsParams::from_grads(loss.backward(), &net);
//! let net = frozen.optimizer().step(1e-2, net, grads);
//! assert_eq!(net.body.bias.as_ref().unwrap().val().into_data(), bias);
//! # Ok::<(), Box<dyn std::error::Error>>(())
//! ```
pub use *;
pub use *;
pub use *;