# 3.1. The Sequential Model
`Sequential` is the container that turns a stack of layers into a trainable model. Construction is a separate type. A caller adds layers in order to a `SequentialBuilder`, then builds it against the shape of the input. That call gives back a `Sequential`, which the caller compiles with an optimizer and a loss before `fit` and `predict`.
3 properties set this model apart. Training is full-batch by default. There is no callback machinery, so early stopping and learning-rate schedules are plain loops. The shape of the input reaches the model 1 time, at `build`, in place of an input layer or a per-layer shape argument.
`SequentialBuilder` and `Sequential` both live in [`rustyml::neural_network::sequential`](https://docs.rs/rustyml). The layers, optimizers, and losses that plug into them each have their own page ([3.2](./3.2._Dense_Layers_and_Activations.md), [3.4](./3.4._Optimizers.md), [3.3](./3.3._Loss_Functions.md)).
## 3.1.1. The lifecycle at a glance
The 6 calls below are the entire public surface that training a network needs. `add` takes the builder by value and gives it back, so a whole stack reads as 1 expression. `build` returns `Result<Sequential, Error>`, and every training call belongs to that built model. `fit` returns `Result<History, Error>`, with 1 loss value per epoch. `predict` returns `Result<Tensor, Error>`.
```rust,ignore
let mut model = SequentialBuilder::new()
.add(/* a layer */)
.add(/* another layer */)
.build(&Shape::known(x.shape()))?; // Result<Sequential, Error>
model.compile(/* optimizer */, /* loss */);
model.summary();
let history = model.fit(&x, &y, epochs)?; // epochs: u32 (history.loss() returns one f32 per epoch)
let y_hat = model.predict(&x_new)?; // Tensor = ArrayD<f32>
```
The [prelude](../Chapter-01/1.5._The_Prelude_and_Imports.md) exports everything on this page. It exports `SequentialBuilder`, `Sequential`, `History`, `Shape`, `Dense`, `Activation`, every optimizer and loss, and the `Tensor` alias. 1 glob is therefore the whole import list of every example below:
```rust,ignore
use rustyml::prelude::*; // SequentialBuilder, Sequential, History, Shape, Dense, Adam, losses, Tensor
```
Every tensor in the framework has type `Tensor`, an alias for `ndarray::ArrayD<f32>`. It has a dynamic rank and always holds `f32` values. Call [`.into_dyn()`](../Chapter-01/1.3._Working_with_ndarray.md) to convert a statically-ranked array, such as `Array2` or `Array3`, into `IxDyn` before it enters a layer. RustyML has no `f64` path. The whole stack uses single precision for cache and SIMD performance.
## 3.1.2. Building the model: `SequentialBuilder` and `build`
Model construction is a separate type. `SequentialBuilder` collects layers, and `Sequential` is a built model. `SequentialBuilder::build` is the only way to reach a `Sequential`. `fit`, `fit_with_batches`, `train_batch`, `evaluate`, `predict`, and `summary` are methods of the built model alone. The weight save and load calls are too. Training a model that was never built is therefore a compile error, and no run-time flag says whether a model is ready.
`SequentialBuilder::new()` creates an empty builder with no layer and no shuffle seed. `add` takes any `L: 'static + Layer` **by value**, boxes it as `Box<dyn Layer>`, and appends it to the stack. `build` takes the shape of the tensor that enters the model:
```rust,ignore
pub fn add<L: 'static + Layer>(self, layer: L) -> Self;
pub fn build(self, input_shape: &Shape) -> Result<Sequential, Error>;
```
`add` consumes the layer, so 1 expression constructs it and moves it: `.add(Dense::new(8, Activation::ReLU).unwrap())`. It also consumes and returns the builder, so the whole stack chains as 1 expression. Layer constructors can fail on their own. An activation with an unusable parameter, for example, returns `Error::InvalidParameter`. That is why `.unwrap()` appears on the layer constructor and not on `add`.
**`add` does no validation, and `build` does all of it.** `add` cannot fail, because nothing about a layer can disagree with the stack until a shape runs through it. `build` walks the stack once from the input. It gives every layer the shape that reaches it, and it threads each output shape into the next layer. A layer that refuses its shape stops the walk, and nothing past that position is allocated. See [3.1.8](#318-errors-shape-checking-and-the-build-time-refusal).
The walk is also where every weight comes from. A layer constructor takes the configuration of the layer and nothing else. A kernel extent comes from the input, and the input is not there yet. `build` is where a layer learns that shape and allocates. A builder therefore holds no weight at all, and a model that reached `Sequential` holds every weight it needs.
### The layer traits, briefly
Using `Sequential` needs no trait implementation at all. Reading the contract explains what `fit` and `predict` call. The framework splits it over 3 traits:
```rust,ignore
// What every layer holds, whatever number of inputs it takes.
pub trait LayerBase: std::any::Any + Send + Sync {
fn layer_type(&self) -> &str;
fn param_count(&self) -> ParamCounts;
fn weights(&self) -> Vec<WeightRef<'_>>; // every array, named, for a checkpoint
fn weights_mut(&mut self) -> Vec<WeightMut<'_>>;
fn parameters_mut(&mut self) -> Vec<ParamRef<'_>>; // the trainable arrays, for the optimizer
fn is_built(&self) -> bool;
fn apply_state(&mut self, state: &mut StateSlot<'_>); // takes back the state a pass proposed
}
// A layer with 1 input, which is almost every layer of the crate.
pub trait UnaryLayer: LayerBase {
fn forward(&self, input: &Tensor, ctx: &mut Ctx) -> Result<Tensor, Error>;
fn backward(&self, grad_output: &Tensor, ctx: &mut Ctx) -> Result<Tensor, Error>;
fn build(&mut self, input: &Shape) -> Result<(), Error>; // allocates, from the shape
fn compute_output_shape(&self, input: &Shape) -> Result<Shape, Error>; // pure, allocates nothing
fn forward_mut(&mut self, input: &Tensor, ctx: &mut Ctx) -> Result<Tensor, Error>;
}
// The general interface, and the one a model holds a layer through.
pub trait Layer: LayerBase {
fn arity(&self) -> Arity; // Exactly(1), AtLeast(2), ...
fn forward_many(&self, inputs: &[&Tensor], ctx: &mut Ctx) -> Result<Tensor, Error>;
fn backward_many(&self, grad: &Tensor, ctx: &mut Ctx) -> Result<Vec<Tensor>, Error>;
fn build_many(&mut self, inputs: &[Shape]) -> Result<(), Error>;
fn compute_output_shape_many(&self, inputs: &[Shape]) -> Result<Shape, Error>;
}
// A layer with 1 input gets the general interface for free.
impl<T: UnaryLayer> Layer for T { /* ... */ }
```
`build` and `compute_output_shape` are the pair that makes a build-time walk possible. `build` allocates, so it takes `&mut self` and runs 1 time per layer. `compute_output_shape` is pure. It reads the layer configuration and the input shape, and it touches no cache that a forward pass wrote.
It answers on a layer that was never built and never run. A `Shape` holds 1 entry per axis, and each entry is either a fixed extent or a free axis. `Layer::output_shape`, which `summary` prints, is `compute_output_shape_many` run against the shapes the layer holds.
There is 1 forward path, and the mode lives in the context rather than in the layer. `forward` takes `&self`, so a layer writes nothing into itself during a pass. `Ctx::training()` builds the context of a training pass, and `Ctx::inference()` builds the context of an inference pass. A training pass parks in the context whatever the backward pass needs. An inference pass parks nothing, and every mode-dependent layer, such as dropout and batch normalization, takes its inference behavior. Because `forward` borrows `&self` and `LayerBase` requires `Send + Sync`, several threads can run inference against 1 model with no lock.
The context has 4 channels. The training flag is the first. The cache stack is the second, and a forward pass writes it for the matching backward pass. The gradient store is the third, and `ctx.grads()` gives every parameter gradient of the pass, keyed by `ParamId::new(scope, name)`.
The state channel is the fourth, and it carries the non-trainable values that a training pass changes. The running statistics of a normalization layer and the random stream of a dropout layer travel there. `LayerBase::apply_state` moves them into the layer after the pass.
`UnaryLayer::forward_mut` is the entry point for a caller that drives 1 layer by hand. It builds the layer from the tensor when the layer holds no build, runs the forward pass, and then applies the proposed state. A model never calls it, because `SequentialBuilder::build` has already built every layer it holds.
Backward propagation is pure math. It does **not** sanitize `NaN` or `Inf` values on purpose. A non-finite gradient propagates and surfaces at the next forward pass or as a `NaN` loss, instead of being masked silently.
## 3.1.3. `compile`: wiring the optimizer and loss
```rust,ignore
pub fn compile<O, LFunc>(&mut self, optimizer: O, loss: LFunc) -> &mut Self
where O: 'static + Optimizer, LFunc: 'static + Loss;
```
`compile` stores the optimizer and loss as trait objects. It returns `&mut Self` so it chains. That is all it does. It runs no shape inference and it allocates nothing, because `build` already gave every layer its arrays.
Training is the only thing that needs `compile`. `fit` returns `Error::NeuralNetwork(NnError::NotCompiled(_))` if the optimizer or the loss is missing. `predict` never reads the optimizer or the loss, so it works on an uncompiled model.
`evaluate` sits between the 2. It needs a loss to score with, and no optimizer to step with. This split matters for serving. After a [weight load](./3.9._Saving_and_Loading_Weights.md) into a fresh architecture, `predict` runs right away without a compiled optimizer.
Pick the optimizer and loss from [3.4](./3.4._Optimizers.md) and [3.3](./3.3._Loss_Functions.md).
Each loss family normalizes its value in a different way. `MeanSquaredError`, `MeanAbsoluteError`, and `BinaryCrossEntropy` average over every element. `CategoricalCrossEntropy` sums over the trailing class axis, then averages over the prediction sites. A prediction site is a sample for a `[batch, classes]` target. For a channels-last convolutional softmax head that predicts 1 class per pixel, a prediction site is a pixel. The divisor is then `batch * height * width`.
A switch between loss families rescales the gradient magnitude, which changes the effective learning rate. Retune the step size after a change of loss.
## 3.1.4. `summary`: reading the architecture
`summary()` prints a table to stdout. Use it to check the wiring and the parameter count before spending epochs on training. Here is the table for a `Dense(2 -> 8, ReLU)` layer followed by a `Dense(8 -> 2, Softmax)` layer:
```text
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ dense (Dense) │ (None, 8) │ 24 │
│ dense_1 (Dense) │ (None, 2) │ 18 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 42 (168 B)
Trainable params: 42 (168 B)
Non-trainable params: 0 (0 B)
```
RustyML generates layer names per type. It names the first `Dense` layer `dense`, the next `dense_1`, and so on. `None` in the output shape is a free axis: the batch axis of a model built with `Shape::with_free_batch`. Build the same stack with `Shape::known(&[4, 2])` instead, and the same 2 rows print `(4, 8)` and `(4, 2)`.
The printed shape is a shape the model produces. `build` gave each layer the shape that reaches it, so every row comes from a shape that the whole stack already agreed on. A model that has never seen a tensor still prints the true output shape of every position. `Sequential::input_shape()` and `Sequential::output_shape()` report the 2 ends of that walk, and `output_shape()` needs no forward pass either.
Parameter counts come from the `param_count()` method of each layer. A `Dense(in -> out)` layer reports `in * out + out` parameters, the weight matrix plus the bias vector. So `2 * 8 + 8 = 24`, and `8 * 2 + 2 = 18`. The byte figures assume 4 bytes per `f32`.
`param_count()` returns a `ParamCounts`, which holds a `trainable` field and a `non_trainable` field. A layer reports both counts at once, so a layer is not trainable or non-trainable as a whole. `summary` adds each field over the model. It prints "Trainable" from the first field, "Non-trainable" from the second, and "Total" from the sum of the 2.
A parameter-free layer, such as an activation or a pooling layer, reports 0 in both fields. `BatchNormalization` reports a non-zero value in both. For `C` channels it holds `2 * C` trainable elements, which are `gamma` and `beta`, and `2 * C` non-trainable elements, which are `moving_mean` and `moving_variance`. Its total is therefore `4 * C`. That count assumes the default settings. `with_center(false)` drops `beta`, and `with_scale(false)` drops `gamma`, so each flag takes `C` off the trainable count and leaves the non-trainable count alone.
`summary` borrows `&self` and never changes the model, so call it at any time, before or after training.
## 3.1.5. `fit` and the training loop
```rust,ignore
pub fn fit(&mut self, x: &Tensor, y: &Tensor, epochs: u32) -> Result<History, Error>;
```
**`fit` is full-batch.** Each epoch runs exactly 1 gradient step over the whole of `x` and `y`. There is only 1 batch, so nothing is shuffled, and the model never reads its shuffle seed. `epochs` counts these full-dataset steps. If 1 gradient step per epoch converges too slowly for the data set, use `fit_with_batches` instead. If a single forward pass does not fit the memory budget, also use it (see below).
Each epoch, `fit` runs 1 `train_batch` call. The [subsection below](#writing-the-loop-by-hand-train_batch-and-evaluate) uses this same public, single-step method to build custom loops. `train_batch` performs these steps, in order:
1. Build a training context with `Ctx::training()`.
2. Forward through every layer, and move the proposed state of each layer into it.
3. Compute the scalar loss.
4. Compute the loss gradient with respect to the output.
5. Advance the global step of the optimizer once, so Adam moves its bias-correction timestep 1 time per step and not 1 time per layer.
6. Backpropagate through the layers in reverse, so each layer adds its parameter gradients to `ctx.grads()`.
7. Apply clip-by-global-norm, if the optimizer asks for it (see [`Optimizer::global_clipnorm`](./3.4._Optimizers.md)).
8. Update the parameters of every layer, from the gradient store.
`step()` runs before the per-layer `update()` calls. This order keeps step-dependent optimizers correct across several layers. The context lives for 1 step alone, so no gradient and no cache can survive into the next step.
`fit` returns a `History`. Its entire API is `loss()`: 1 `f32` per epoch, in epoch order. `epochs = 0` gives an empty slice. Each entry is the mean per-sample loss measured **during** the epoch, before the update of that epoch. Every batch contributes the loss from the forward pass that ran before its own weight update. Each entry therefore describes the weights the model held while the epoch ran, and never the weights the epoch ends with.
Reading the last entry as the final loss of the trained model is wrong in both directions. While training converges, the entry reads above the truth, because the updates of that epoch already improved on the weights it measured. Once the step size overshoots, the entry reads below the truth, because those same updates made things worse. `evaluate` (see below) reports the loss of the model in hand.
The `show_progress` feature only adds a display. It is not the only way to read the loss. When it is on, `fit` renders a live progress bar. The bar shows the loss of the current epoch with 6 decimal places. `fit_with_batches` renders a similar bar that tracks the running average loss as its batches complete:
```toml
[dependencies]
rustyml = { version = "0.16", features = ["neural_network", "show_progress"] }
```
```text
[00:00:00] ████████████████████████████████████████ 400/400 | Loss: <current loss>
```
Without the feature, training runs silently. The returned `History` holds the same numbers either way, so no caller depends on whether the feature is on.
### Mini-batch training with `fit_with_batches`
```rust,ignore
pub fn fit_with_batches(&mut self, x: &Tensor, y: &Tensor, epochs: u32, batch_size: usize)
-> Result<History, Error>;
```
`fit_with_batches` is the mini-batch loop. It reshuffles the sample order at the start of every epoch, then trains on fixed-size chunks. The shuffle makes this the 1 place where the seed of the model matters. Set it with `SequentialBuilder::new_with_seed(seed)` or `Sequential::set_seed(seed)` for a reproducible shuffle order (see [7.1](../Chapter-07/7.1._Reproducibility_and_Random_Seeds.md)). This seed governs the shuffle alone. It does not affect weight initialization, which each layer seeds through its own `with_random_state` call.
A `batch_size` of `0`, or one larger than the data set, returns `Error::InvalidParameter`. A `batch_size` equal to `n_samples` degenerates to a single full-batch step per epoch, which matches `fit`.
Its `History` entries mean what the entries of `fit` mean, except for 1 refinement. That refinement shows only when `batch_size` does not divide the data set evenly. Each batch contributes to the epoch loss in proportion to its sample count, and not as 1 vote per batch. A short trailing batch therefore pulls the epoch figure less than a full batch does. This makes each entry exactly the mean per-sample loss over the whole data set. A plain mean over batches would not give that number.
```rust
use ndarray::Array;
use rustyml::prelude::*;
fn main() {
let x = Array::ones((8, 3)).into_dyn();
let y = Array::zeros((8, 1)).into_dyn();
// Seed the per-epoch shuffle so the run is reproducible.
let mut model = SequentialBuilder::new_with_seed(0)
.add(Dense::new(6, Activation::ReLU).unwrap())
.add(Dense::new(1, Activation::Sigmoid).unwrap())
.build(&Shape::known(x.shape()))
.unwrap();
model.compile(
SGD::new(0.05, 0.9, false, 0.0).unwrap(),
BinaryCrossEntropy::new(),
);
// 4 mini-batches of 2 samples per epoch, reshuffled every epoch.
let history = model.fit_with_batches(&x, &y, 5, 2).unwrap();
assert_eq!(history.loss().len(), 5);
// External LR schedule: read the step size, halve it, write it back. The optimizer keeps
// its momentum buffers across the change.
let lr = model.learning_rate().unwrap();
model.set_learning_rate(lr * 0.5);
let resumed = model.fit_with_batches(&x, &y, 5, 2).unwrap();
// Training resumes where it left off rather than restarting.
assert!(resumed.loss()[0] < *history.loss().last().unwrap());
assert_eq!(model.predict(&x).unwrap().shape(), &[8, 1]);
}
```
`set_learning_rate`, used above, is the hook for external schedules such as step decay or warmup. It retunes the step size in place. The optimizer keeps all of its accumulated state, such as momentum buffers and Adam moments, across the change. It does nothing on a model that was never compiled.
`learning_rate()` is its read half. It returns `None` on an uncompiled model. This read-scale-write pattern avoids a second copy of the rate beside the model, which could drift out of sync with the optimizer. Unlike the optimizer constructors, `set_learning_rate` validates nothing. `learning_rate()` returns exactly the value that was set, even a zero or a negative rate.
### Writing the loop by hand: `train_batch` and `evaluate`
```rust,ignore
pub fn train_batch(&mut self, x: &Tensor, y: &Tensor) -> Result<f32, Error>;
pub fn evaluate(&self, x: &Tensor, y: &Tensor) -> Result<f32, Error>;
```
`train_batch` is the single step that both `fit` variants build on. It is public. Any other epoch structure is a loop the caller writes, instead of a fork of the library. Examples include curriculum ordering, a per-step schedule, or a probe between steps.
The whole of `x` is the batch. Nothing is split, nothing is shuffled, and mode-dependent layers run in training mode. The returned `f32` is the loss from the forward pass, measured before the update of that call. This is the number `fit` records for each epoch. It is also the value that `fit_with_batches` averages, weighted by sample count, into each `History` entry.
`train_batch` validates its own inputs instead of trusting the caller. A call on an uncompiled model therefore returns a `NotCompiled` error, and not a panic.
`evaluate` is the other half. It runs 1 inference-mode forward pass over the whole of `x` and scores it with the compiled loss. It updates nothing: no gradients, no parameters, and no batch-norm running statistics. It borrows `&self`, so scoring a model between training steps cannot disturb it. It also draws from no random number generator, so a call inside a `fit_with_batches` loop cannot perturb the shuffle stream.
Layers behave exactly as they do in `predict`. Dropout and noise layers act as the identity, and batch normalization reads its running statistics. On a model with such layers, `evaluate` and the number `fit` recorded for the same data therefore disagree. Training-mode dropout inflates the number `fit` records. `evaluate` gives the more accurate estimate of the 2.
Together, `train_batch` and `evaluate` turn early stopping, learning-rate schedules, and checkpoint selection into ordinary code. The loop below takes 1 full-batch step at a time. It scores the model it holds after each step. It halves the step size after 10 steps without an improvement, and it stops once the schedule runs out:
```rust
use ndarray::Array;
use rustyml::prelude::*;
fn main() {
// 8 points of y = 2x + 1.
let x = Array::from_shape_vec((8, 1), (0..8).map(|i| i as f32 / 8.0).collect::<Vec<_>>())
.unwrap()
.into_dyn();
let y = x.mapv(|v| 2.0 * v + 1.0);
let mut model = SequentialBuilder::new()
.add(Dense::new(8, Activation::Tanh).unwrap().with_random_state(0))
.add(Dense::new(1, Activation::Linear).unwrap().with_random_state(0))
.build(&Shape::known(x.shape()))
.unwrap();
model.compile(SGD::new(0.1, 0.9, false, 0.0).unwrap(), MeanSquaredError::new());
let start = model.evaluate(&x, &y).unwrap();
let (mut best, mut stale, mut previous) = (start, 0, start);
for _ in 0..500 {
// The step reports the loss it started from. This stack has no dropout, so that
// number is the previous `evaluate` measured again, one step stale.
let during = model.train_batch(&x, &y).unwrap();
assert!((during - previous).abs() < 1e-5);
// This one scores the weights the step produced.
let after = model.evaluate(&x, &y).unwrap();
previous = after;
if after < best - 1e-6 {
best = after;
stale = 0;
continue;
}
// 10 steps without progress: halve the step size, read from the optimizer instead of
// a copy kept here. Once it is that small, there is nothing left to try.
stale += 1;
if stale == 10 {
let lr = model.learning_rate().unwrap();
if lr < 1e-4 {
break;
}
model.set_learning_rate(lr * 0.5);
stale = 0;
}
}
assert!(best < start / 100.0);
}
```
## 3.1.6. `predict`: forward-only inference
```rust,ignore
pub fn predict(&self, x: &Tensor) -> Result<Tensor, Error>;
```
`predict` builds an inference context with `Ctx::inference()`, runs every layer against it, and returns the output `Tensor`. It borrows `&self`, it writes no backward cache, and it puts [mode-dependent layers](./3.8._Regularization_and_Normalization_Layers.md) into inference behavior. Dropout is disabled, and batch normalization uses its running statistics, which is the correct behavior for serving. `predict` is deterministic. 2 calls on the same input return identical tensors. Unlike training and evaluation, `predict` does not need `compile`.
The result shape is whatever the last layer emits, `(batch, units)` for a `Dense` tail on a rank-2 input. `Sequential::output_shape()` reports it without a forward pass, because it runs the same pure shape walk `build` ran.
The input must agree with the shape the model was built for, on every axis except the batch axis. A built layer refuses an input that disagrees, and it accepts any batch size, so a partial final mini-batch always passes. A `Dense` layer built for a last axis of 8 refuses a tensor of rank 1. It refuses a last axis of any other width as well, with `Error::InvalidInput`.
## 3.1.7. A complete example: learning XOR
XOR is the smallest problem that a linear model cannot solve. It is the classic proof that a hidden layer does real work. This example uses 2 `Dense` layers: a `Tanh` hidden layer and a `Softmax` head over 2 classes. Training with Adam and categorical cross-entropy separates the classes cleanly.
The targets are one-hot: class 0 is `[1, 0]`, and class 1 is `[0, 1]`. A seed of `with_random_state(0)` on both layers makes the run reproducible. The builder only records that seed, and `build` spends it, so the layer draws its arrays 1 time.
```rust
use ndarray::Array;
use rustyml::prelude::*;
fn main() {
// XOR inputs, shape (4, 2).
let x = Array::from_shape_vec((4, 2), vec![0.0_f32, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0])
.unwrap()
.into_dyn();
// One-hot targets: XOR is class 1 for (0,1) and (1,0), class 0 otherwise.
let y = Array::from_shape_vec((4, 2), vec![1.0_f32, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0])
.unwrap()
.into_dyn();
let mut model = SequentialBuilder::new()
.add(Dense::new(8, Activation::Tanh).unwrap().with_random_state(0))
.add(
Dense::new(2, Activation::Softmax { axis: -1 })
.unwrap()
.with_random_state(0),
)
.build(&Shape::with_free_batch(x.shape()))
.unwrap();
model.compile(
Adam::new(0.1, 0.9, 0.999, 1e-8, 0.0).unwrap(),
CategoricalCrossEntropy::new(false),
);
model.summary();
let history = model.fit(&x, &y, 400).unwrap();
assert!(*history.loss().last().unwrap() < history.loss()[0] / 100.0);
let preds = model.predict(&x).unwrap();
for i in 0..4 {
// argmax over the 2 class probabilities
let class = if preds[[i, 0]] >= preds[[i, 1]] { 0 } else { 1 };
println!("row {i} -> class {class} probs [{:.3}, {:.3}]", preds[[i, 0]], preds[[i, 1]]);
}
}
```
After 400 full-batch epochs, the network assigns the 2 true XOR rows to class 1 and the 2 false rows to class 0. Each probability sits at or near `1.000`. The `History` records that descent. Its last epoch sits more than 2 orders of magnitude below its first, which is what the assertion checks. The `CategoricalCrossEntropy::new(false)` argument tells the loss that the head already produces probabilities, from the `Softmax` layer, so the loss does not apply its own log-softmax. When the last layer emits raw logits, pass `true` instead.
## 3.1.8. Errors, shape checking, and the build-time refusal
`fit`, `fit_with_batches`, and `train_batch` validate the model and the inputs before they touch any layer. The checks run in this order:
1. An optimizer is present.
2. A loss is present.
3. The model has at least 1 layer.
4. The inputs have a batch axis.
5. The inputs are not empty.
6. `x` and `y` agree on batch size.
`evaluate` runs the same checks, except for the optimizer check, because only a parameter update needs an optimizer. The table below maps each failure to its error variant. All of them are [`rustyml::error::Error`](../Chapter-01/1.6._Error_Handling.md) values:
| Situation | Returned error |
|---|---|
| `fit`/`fit_with_batches`/`train_batch` before `compile` | `Error::NeuralNetwork(NnError::NotCompiled("optimizer"))` |
| `evaluate` before `compile` | `Error::NeuralNetwork(NnError::NotCompiled("loss function"))` |
| `build` on a builder that holds no layer | `Error::NeuralNetwork(NnError::EmptyModel)` |
| rank-0 `x` or `y` (a scalar tensor, so no batch axis) | `Error::InvalidInput(_)` |
| empty `x` or `y` | `Error::EmptyInput(_)` |
| `x` and `y` disagree on batch size (rows) | `Error::DimensionMismatch { .. }` |
| `fit_with_batches` with `batch_size == 0` or `> n_samples` | `Error::InvalidParameter { .. }` |
| a layer that refuses the shape `build` gives it | `Error::InvalidInput(_)`, naming the position and the type |
The rank-0 row needs an explanation. A 0-dimensional tensor is not an empty one. It holds exactly 1 element, so `is_empty` does not reject it, and the batch-axis index that follows has nothing to read. That case returns `InvalidInput`.
These are recoverable `Result` values. The next example shows the `compile` requirement. It shows how training and evaluation each demand something different from `compile`, and how `predict` needs neither:
```rust
use ndarray::Array;
use rustyml::error::Error;
use rustyml::neural_network::NnError;
use rustyml::prelude::*;
fn main() {
let x = Array::ones((4, 2)).into_dyn();
let y = Array::ones((4, 1)).into_dyn();
let mut model = SequentialBuilder::new()
.add(Dense::new(1, Activation::Linear).unwrap())
.build(&Shape::known(x.shape()))
.unwrap();
// fit needs an optimizer and a loss. Without compile, it fails fast and names the first
// thing it found missing.
match model.fit(&x, &y, 1) {
Err(Error::NeuralNetwork(NnError::NotCompiled(missing))) => assert_eq!(missing, "optimizer"),
other => panic!("expected NotCompiled, got {other:?}"),
}
// evaluate updates nothing, so it asks only for the loss it scores with.
match model.evaluate(&x, &y) {
Err(Error::NeuralNetwork(NnError::NotCompiled(missing))) => {
assert_eq!(missing, "loss function")
}
other => panic!("expected NotCompiled, got {other:?}"),
}
// predict, by contrast, never needs compile: it is forward-only.
assert_eq!(model.predict(&x).unwrap().shape(), &[4, 1]);
// A builder that holds no layer has nothing to build.
match SequentialBuilder::new().build(&Shape::known(&[4, 2])) {
Err(Error::NeuralNetwork(NnError::EmptyModel)) => {}
_ => panic!("expected EmptyModel"),
}
}
```
A stack whose shapes do not agree is the last case. **`build` refuses a shape mismatch before any data moves, and the message names the layer.** No adjacent pair can disagree in silence, because a layer no longer declares the width it expects. `build` gives each layer the shape that reaches it, and a layer that cannot accept that shape stops the walk. The example below asks a 4-by-4 pooling window to run over a 3-by-3 feature map:
```rust
use rustyml::prelude::*;
fn main() {
// A 3x3 valid convolution over a 5x5 image emits 3x3, and a 4x4 pooling window does not
// fit in it. `build` walks the stack, so the refusal arrives here.
let refused = SequentialBuilder::new()
.add(Conv2D::new(4, (3, 3), (1, 1), Activation::ReLU).unwrap())
.add(MaxPooling2D::new((4, 4)))
.add(Flatten::new())
.add(Dense::new(2, Activation::Linear).unwrap())
.build(&Shape::with_free_batch(&[8, 5, 5, 1]));
let message = match refused {
Ok(_) => panic!("the 4x4 window does not fit"),
Err(error) => error.to_string(),
};
assert!(message.contains("layer 1"), "{message}");
assert!(message.contains("MaxPooling2D"), "{message}");
println!("{message}");
}
```
```text
invalid input: layer 1 (`MaxPooling2D`) refused the input shape (None, 3, 3, 4): invalid
parameter `pool_size`: cannot exceed the corresponding input dimension
```
Read the 3 parts of that message. `layer 1` is the position in the stack, counted from the input. `MaxPooling2D` is the type at that position. `(None, 3, 3, 4)` is the shape that actually arrived, which the convolution in front of it produced. The 3 parts together point at 1 line of the model.
This is the reason the shape reaches the model at `build`. The same mistake used to surface in the middle of a forward pass, deep inside a model, and with no layer named. It arrived after the first epoch had already started. It now surfaces on the line that assembles the model, and nothing past the layer at fault is ever allocated. `Sequential::output_shape()` answers the same question in the other direction, and it needs no data either.
Each building block has its own page from here. See [Dense layers and activations](./3.2._Dense_Layers_and_Activations.md), [loss functions](./3.3._Loss_Functions.md), [optimizers](./3.4._Optimizers.md), the convolutional and recurrent layers, and [saving and loading weights](./3.9._Saving_and_Loading_Weights.md). For a model that a chain cannot express, see [graph models and merge layers](./3.10._Graph_Models_and_Merge_Layers.md).