1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
//! Parameter-group optimizers: several optimizers, and several learning
//! rates, over one model.
//!
//! A `burn` [`Optimizer`](burn::optim::Optimizer) steps every parameter of a
//! module with one rule. Training recipes often want more than that:
//!
//! - 2-D weight matrices train well with [`Muon`](burn::optim::Muon), while
//! embeddings, norms and biases are better served by
//! [`AdamW`](burn::optim::AdamW);
//! - groups want their own learning rates: the `NanoChat` recipe scales the
//! `lm_head` and embedding rates separately, by a `d_model` factor;
//! - groups want their own weight decay or betas.
//!
//! The `GroupOptimizerAdaptorN` family is a single
//! [`Optimizer`](burn::optim::Optimizer) that mounts `N` *types* of
//! optimizer, each with any number of parameter groups, and steps each
//! parameter with the group that claims it.
//!
//! This module is behind the `train` feature, which is on by default.
//!
//! # Lifecycle
//!
//! 1. **Select `ParamId`s.** Choose each group's parameters by structure with
//! [`module::reflection`](crate::burner::module::reflection): build an
//! [`XmlModuleTree`](crate::burner::module::reflection::XmlModuleTree) over
//! the live module and select from it with `XPath`
//! ([`select_param_ids`](crate::burner::module::reflection::XmlModuleTree::select_param_ids)).
//! 2. **Build [`OptimizerGroup`]s.** A group is a set of `ParamId`s, the
//! optimizer that steps them, and an optional [`LrSelector`].
//! [`OptimizerGroup::from_adaptor`] takes the optimizer from what a `burn`
//! optimizer config's `init()` returns. [`OptimizerGroup::frozen`] builds a
//! group whose parameters are never moved ([`FrozenOptimizer`]).
//! 3. **Compose with `GroupOptimizerAdaptorN::new`.** Pass the module, then one
//! `Vec<OptimizerGroup<B, Oi>>` per optimizer type; e.g.
//! [`GroupOptimizerAdaptor2::new`] takes two. It checks that the groups
//! partition the module's float parameters: none in two groups, none in no
//! group. `new_with_policy` (e.g.
//! [`GroupOptimizerAdaptor2::new_with_policy`]) takes an
//! [`UnknownParamPolicy`] after the module; `new` uses the default, `Panic`.
//! 4. **Train.** The adaptor implements [`Optimizer`](burn::optim::Optimizer),
//! so it goes wherever a single optimizer would: a
//! [`Learner`](burn::train::Learner), or your own loop calling `step`. Step
//! the module whose `ParamId`s the groups hold: `step` applies the policy to
//! any float parameter it doesn't know. To resume from a checkpoint, build
//! the same way and load the records: see [Resuming from a
//! checkpoint](#resuming-from-a-checkpoint).
//!
//! [`GroupOptimizerAdaptor1`] through [`GroupOptimizerAdaptor7`] exist. Pick
//! the smallest `N` that covers your optimizer *types*: more groups of one
//! type go in that type's `Vec`, and don't raise `N`.
//! `GroupOptimizerAdaptor1` is "one optimizer type, several learning-rate
//! groups".
//!
//! # Resuming from a checkpoint
//!
//! `ParamId`s persist across a checkpoint. A module record saves each
//! parameter's id, and `Module::load_record` gives the parameter that id
//! back, so a restored model has the saved run's ids, not the ones it was
//! built with. The adaptor's record, a [`GroupOptimizerAdaptorRecord`], saves
//! which group steps each id next to the groups' state, and the adaptor's
//! `load_record` restores both. Load the model record and the optimizer record
//! saved with it, in either order, and the adaptor knows the restored model's
//! ids.
//!
//! So a resumed run is built like a first run:
//!
//! 1. Build a fresh model, select its groups, and build the adaptor, with the
//! same groups in the same order as the run that saved the checkpoint. The
//! fresh ids don't matter once the records are loaded.
//! 2. Load the checkpoint into both. A [`Learner`](burn::train::Learner)
//! resuming from a checkpoint does this: it loads the model record, then the
//! optimizer record. In your own loop, call `Module::load_record` and
//! `Optimizer::load_record`.
//!
//! What the adaptor's `load_record` restores, and what it keeps:
//!
//! - **The record's groups replace the ones `new` built.** Groups are matched
//! by position, `(optimizer type, group index)`, and each keeps the optimizer
//! and learning rate it was given at `new`. `load_record` panics when the
//! record doesn't fit: another number of groups of some optimizer type, a
//! group the adaptor doesn't have, or a `ParamId` the adaptor knows in
//! another group. The last happens when the groups were selected from an
//! already restored model, and the selection changed since the save.
//! - **The [`UnknownParamPolicy`] still applies.** A float parameter the record
//! doesn't name, such as a head added since the save, is unknown at `step`.
//!
//! ```rust
//! use bunsen::{
//! burner::optim::{
//! GroupOptimizerAdaptor1,
//! GroupOptimizerError,
//! OptimizerGroup,
//! },
//! support::testing::{
//! CpuBackend,
//! default_device,
//! },
//! };
//! use burn::{
//! backend::Autodiff,
//! module::Module,
//! nn::{
//! Linear,
//! LinearConfig,
//! },
//! optim::{
//! GradientsParams,
//! Optimizer,
//! Sgd,
//! SgdConfig,
//! },
//! prelude::Device,
//! tensor::Tensor,
//! };
//!
//! type B = Autodiff<CpuBackend>;
//! type Optim = GroupOptimizerAdaptor1<Sgd<CpuBackend>, Linear<B>, B>;
//!
//! /// Builds the model and its adaptor: the same code starts a run and resumes
//! /// one.
//! fn build(
//! device: &Device<B>
//! ) -> Result<(Linear<B>, Optim), GroupOptimizerError> {
//! let net: Linear<B> = LinearConfig::new(4, 2).init(device);
//! let params = [net.weight.id, net.bias.as_ref().unwrap().id];
//! let sgd = SgdConfig::new().init::<B, Linear<B>>();
//! let optim =
//! Optim::new(&net, vec![OptimizerGroup::from_adaptor(params, &sgd)])?;
//! Ok((net, optim))
//! }
//!
//! fn train_step(
//! net: Linear<B>,
//! optim: &mut Optim,
//! device: &Device<B>,
//! ) -> Linear<B> {
//! let loss = net.forward(Tensor::<B, 2>::ones([3, 4], device)).sum();
//! let grads = GradientsParams::from_grads(loss.backward(), &net);
//! optim.step(1e-2, net, grads)
//! }
//!
//! let device = default_device();
//!
//! // The first run trains, and saves a checkpoint.
//! let (net, mut optim) = build(&device)?;
//! let net = train_step(net, &mut optim, &device);
//! let (model_record, optim_record) = (net.into_record(), optim.to_record());
//!
//! // The resumed run builds a fresh model, with fresh `ParamId`s, and an
//! // adaptor over it, then loads both records.
//! let (net, optim) = build(&device)?;
//! let net = net.load_record(model_record);
//! let mut optim = optim.load_record(optim_record);
//!
//! // The adaptor knows the restored ids, so `step` steps them.
//! let before = net.weight.val().into_data();
//! let net = train_step(net, &mut optim, &device);
//! assert_ne!(net.weight.val().into_data(), before);
//! # Ok::<(), Box<dyn std::error::Error>>(())
//! ```
//!
//! # Behaviour to know
//!
//! - **Duplicates are rejected.** A `ParamId` in two groups, of the same
//! optimizer type or not, makes `new` return
//! [`GroupOptimizerError::DuplicateParamId`].
//! - **Every float parameter needs a group.** A float parameter of the module
//! in no group makes `new` return
//! [`GroupOptimizerError::UnassignedParamIds`], listing them, under the
//! default policy (below). This includes a parameter with `require_grad` off,
//! which gets no gradient and so is not stepped whatever its group. Put
//! parameters you mean to keep fixed in a frozen group
//! ([`OptimizerGroup::frozen`]); a frozen group is an optimizer type of its
//! own, so it takes one of the `N`. Build a remnant group from the parameters
//! no other group claims.
//! - **`step` checks the module it is given.** The groups hold `ParamId`s, and
//! a float parameter whose `ParamId` the adaptor doesn't know is never
//! stepped: its gradient is dropped. That happens when the module changed
//! after `new` (surgery, a fresh head), when `step` gets another module, or
//! when a module record was loaded without the optimizer record saved with
//! it: `Module::load_record` gives the parameters the record's ids. Select
//! the groups after surgery. To load a checkpoint, load both records; see
//! [Resuming from a checkpoint](#resuming-from-a-checkpoint).
//! - **The [`UnknownParamPolicy`] decides what a parameter in no group does**,
//! at `new` and at `step`. `Panic`, the default, fails: `new` returns
//! `UnassignedParamIds`, and `step` panics before it steps anything, naming
//! the ids. `Warn` logs each such `ParamId` once, through the `log` crate,
//! and leaves the parameter unchanged; `Freeze` leaves it unchanged silently.
//! Under both, `new` accepts the module.
//! - **Empty groups are not errors.** An `XPath` selection that matches nothing
//! gives an empty group, and the remnant group then claims the parameters it
//! meant to, with the remnant's settings. Assert that each group is
//! non-empty.
//! - **Only float parameters are stepped.** Int and bool parameters keep their
//! values, whatever group they are in, and need no group.
//! - **Gradient clipping**, set with `with_grad_clipping` (e.g.
//! [`GroupOptimizerAdaptor2::with_grad_clipping`]), applies to each
//! parameter's gradient on its own, before its group's step: a norm clip
//! bounds each tensor's norm, not a global norm.
//! - **The record holds the groups and their state.** A
//! [`GroupOptimizerAdaptorRecord`] holds which group steps each `ParamId`
//! (`dispatch`), and the state: one `Vec<OptimizerGroupRecord>` per optimizer
//! type, one entry per group, each keyed by `ParamId`. Load it into an
//! adaptor built with the same groups, in the same order; `load_record`
//! panics if they don't fit.
//!
//! # Learning rates
//!
//! Each step receives a *global* learning rate: the `lr` passed to `step`,
//! which a `Learner` takes from its scheduler. Each group maps it through its
//! [`LrSelector`]:
//!
//! - no selector, or [`GlobalLrSelector`]: the global rate;
//! - [`OptimizerGroup::with_fixed_lr`] ([`FixedLrSelector`]): a constant that
//! ignores the schedule;
//! - a closure, e.g. a per-group factor on the scheduled rate, as in the
//! example below. A `Send + Sync` closure from the global rate to the group's
//! rate is an `LrSelector`.
//!
//! # Example
//!
//! Muon for the matrices, `AdamW` at half the scheduled rate for the biases:
//!
//! ```rust
//! use bunsen::{
//! burner::{
//! module::reflection::XmlModuleTree,
//! optim::{
//! FrozenOptimizer,
//! GroupOptimizerAdaptor1,
//! GroupOptimizerAdaptor2,
//! GroupOptimizerError,
//! OptimizerGroup,
//! },
//! },
//! public::hashbrown::HashSet,
//! };
//! use burn::{
//! backend::Autodiff,
//! module::{
//! Module,
//! ParamId,
//! },
//! nn::{
//! Linear,
//! LinearConfig,
//! },
//! optim::{
//! AdamWConfig,
//! GradientsParams,
//! MuonConfig,
//! Optimizer,
//! },
//! prelude::Backend,
//! tensor::Tensor,
//! };
//!
//! #[derive(Module, Debug)]
//! struct Net<B: Backend> {
//! body: Linear<B>,
//! head: Linear<B>,
//! }
//!
//! type B = Autodiff<bunsen::support::testing::CpuBackend>;
//! let device = bunsen::support::testing::default_device();
//! let net: Net<B> = Net {
//! body: LinearConfig::new(4, 4).init(&device),
//! head: LinearConfig::new(4, 2).init(&device),
//! };
//!
//! // 1. Select. Elements are type names (`Net`, `Linear`); fields are `@name`.
//! let mut mtree = XmlModuleTree::build(&net);
//! let matrices: HashSet<ParamId> = mtree
//! .select_param_ids("Net/Linear/*[@name='weight'][@rank=2]")?
//! .into_iter()
//! .collect();
//! let rest: HashSet<ParamId> = mtree
//! .param_ids()?
//! .into_iter()
//! .filter(|id| !matrices.contains(id))
//! .collect();
//! assert_eq!((matrices.len(), rest.len()), (2, 2));
//!
//! // 2. Groups.
//! let muon = MuonConfig::new().init::<B, Net<B>>();
//! let adamw = AdamWConfig::new().init::<B, Net<B>>();
//! let matrix_group = OptimizerGroup::from_adaptor(matrices.clone(), &muon);
//! let rest_group = OptimizerGroup::from_adaptor(rest.clone(), &adamw)
//! .with_lr_selector(|lr| lr * 0.5);
//!
//! // 3. Compose: the module, then one `Vec` of groups per optimizer type.
//! let mut optim: GroupOptimizerAdaptor2<_, _, Net<B>, B> =
//! GroupOptimizerAdaptor2::new(&net, vec![matrix_group], vec![rest_group])?;
//!
//! // 4. Step it like any `burn` optimizer, or hand it to a `Learner`.
//! let before = net.body.weight.val().into_data();
//! let x = Tensor::<B, 2>::ones([3, 4], &device);
//! let loss = net.head.forward(net.body.forward(x)).sum();
//! let grads = GradientsParams::from_grads(loss.backward(), &net);
//! let net = optim.step(1e-2, net, grads);
//! assert_ne!(net.body.weight.val().into_data(), before);
//!
//! // A parameter claimed by two groups is rejected.
//! let twice: Result<GroupOptimizerAdaptor2<_, _, Net<B>, B>, _> =
//! GroupOptimizerAdaptor2::new(
//! &net,
//! vec![OptimizerGroup::from_adaptor(matrices.clone(), &muon)],
//! vec![OptimizerGroup::from_adaptor(mtree.param_ids()?, &adamw)],
//! );
//! assert!(matches!(
//! twice,
//! Err(GroupOptimizerError::DuplicateParamId { .. })
//! ));
//!
//! // So is a parameter claimed by none: here, the biases.
//! let partial: Result<GroupOptimizerAdaptor1<_, Net<B>, B>, _> =
//! GroupOptimizerAdaptor1::new(
//! &net,
//! vec![OptimizerGroup::from_adaptor(matrices.clone(), &muon)],
//! );
//! assert!(matches!(
//! partial,
//! Err(GroupOptimizerError::UnassignedParamIds { param_ids }) if param_ids.len() == 2
//! ));
//!
//! // To keep the biases fixed, freeze them in a group of their own.
//! let mut frozen: GroupOptimizerAdaptor2<_, FrozenOptimizer, Net<B>, B> =
//! GroupOptimizerAdaptor2::new(
//! &net,
//! vec![OptimizerGroup::from_adaptor(matrices, &muon)],
//! vec![OptimizerGroup::frozen(rest)],
//! )?;
//! let bias = net.body.bias.as_ref().unwrap().val().into_data();
//! let x = Tensor::<B, 2>::ones([3, 4], &device);
//! let loss = net.head.forward(net.body.forward(x)).sum();
//! let grads = GradientsParams::from_grads(loss.backward(), &net);
//! let net = frozen.step(1e-2, net, grads);
//! assert_eq!(net.body.bias.as_ref().unwrap().val().into_data(), bias);
//! # Ok::<(), Box<dyn std::error::Error>>(())
//! ```
pub use *;
pub use *;
pub use *;