# 3.5. Convolutional Layers
RustyML ships 10 convolutional layers. `Conv1D`, `Conv2D`, and `Conv3D` are the plain layers. `Conv1DTranspose`, `Conv2DTranspose`, and `Conv3DTranspose` are their 3 transposed counterparts. `DepthwiseConv1D`, `DepthwiseConv2D`, `SeparableConv1D`, and `SeparableConv2D` are the depthwise and separable pairs. All 10 live in `rustyml::neural_network::layers`, and the layers glob re-exports them.
They share 1 design. A layer struct holds the weights, the bias, the activation, and the forward and backward caches. The layer delegates the actual numerics for the plain convolutions to a single dimension-generic engine.
If you know Keras, you already know the layouts and weight shapes here: tensors are channels-last, and kernels carry their taps first. 1 difference catches new users early. You pass the full input shape into the constructor. This lets the layer size its weights up front, instead of inferring them lazily on the first batch.
This page covers the layouts, the constructor and padding rules, and the im2col plus GEMM engine behind the standard convolutions. It also covers the depthwise and separable factorization with its cost math, and the transposed convolutions that run a convolution backwards to grow a tensor. It then covers the border layers that resize the spatial axes by hand. The last 3 sections cover the layers that reorder axes or repeat a vector, the layer that does nothing at all, and the error types.
## 3.5.1. Tensor Layouts and the Layer Family
Every convolution here is channels-last. The spatial axes follow the batch axis, and the channel axis comes last. A `Conv2D` reads `[batch, height, width, channels]` and writes `[batch, out_height, out_width, filters]`. This is the Keras and TensorFlow `NHWC` convention.
A tensor built for Keras needs no permutation. If you build inputs by hand with `ndarray`, put the channels last. The weight tensor uses Keras' kernel shape too: the kernel taps come first, then the input channels, then the filters.
That is not only an interface choice. With the channel axis innermost, a kernel tap at a given output position is `Cin` contiguous floats. This lets the engine build its im2col matrix from run copies, instead of a scalar gather. It also lets the flat weight matrix line up with the im2col matrix without any permutation.
| Layer | Input tensor | Weight tensor | Bias | Output tensor |
| --- | --- | --- | --- | --- |
| `Conv1D` | `[N, L, Cin]` | `[k, Cin, F]` | `[F]` | `[N, L', F]` |
| `Conv2D` | `[N, H, W, Cin]` | `[kh, kw, Cin, F]` | `[F]` | `[N, H', W', F]` |
| `Conv3D` | `[N, D, H, W, Cin]` | `[kd, kh, kw, Cin, F]` | `[F]` | `[N, D', H', W', F]` |
| `Conv1DTranspose` | `[N, L, Cin]` | `[k, F, Cin]` | `[F]` | `[N, L', F]` |
| `Conv2DTranspose` | `[N, H, W, Cin]` | `[kh, kw, F, Cin]` | `[F]` | `[N, H', W', F]` |
| `Conv3DTranspose` | `[N, D, H, W, Cin]` | `[kd, kh, kw, F, Cin]` | `[F]` | `[N, D', H', W', F]` |
| `DepthwiseConv1D` | `[N, L, C]` | `[k, C, dm]` | `[C*dm]` | `[N, L', C*dm]` |
| `DepthwiseConv2D` | `[N, H, W, C]` | `[kh, kw, C, dm]` | `[C*dm]` | `[N, H', W', C*dm]` |
| `SeparableConv1D` | `[N, L, Cin]` | depthwise `[k, Cin, dm]`, pointwise `[1, Cin*dm, F]` | `[F]` | `[N, L', F]` |
| `SeparableConv2D` | `[N, H, W, Cin]` | depthwise `[kh, kw, Cin, dm]`, pointwise `[1, 1, Cin*dm, F]` | `[F]` | `[N, H', W', F]` |
Every shape in the table matches Keras, so a kernel exported from Keras drops in without a permutation. Note that the 3 transposed kernels carry their filter axis before their input-channel axis, which is the reverse of the plain ones. Section 3.5.5 explains why.
The 4 depthwise and separable layers need more explanation. A depthwise layer has no `filters` argument at all. It applies `depth_multiplier` kernels to each input channel and emits `C * dm` channels. Input channel `c`'s multiplier `m` lands at output channel `c * dm + m`.
A separable layer holds 2 weight tensors because it fuses 2 convolutions into 1 layer. The depthwise stage emits its channels in that same `c * dm + m` order. This means the pointwise weight's rows already match the depthwise output, so the layer needs no repacking between the stages. Section 3.5.4 covers this in more detail.
The forward math is cross-correlation, the same convention Keras and PyTorch use. The engine does not flip the kernel. The bias is added last, once per filter, after the multiply-accumulate step. Weights start from Xavier/Glorot uniform bounds. Biases start at zero.
The following example sets the weights by hand and runs 1 forward pass. It shows the layout and the Valid-padding output size.
```rust
use ndarray::{Array, Array1, Array4};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::Layer;
fn main() {
// Channels-last: [batch, height, width, channels].
let mut layer = Conv2D::new(1, (2, 2), vec![1, 4, 4, 1], (1, 1), Activation::Linear).unwrap();
// Weights are [kernel_h, kernel_w, channels, filters]. Bias is [filters].
let weights = Array4::from_elem((2, 2, 1, 1), 1.0f32);
let bias = Array1::zeros(1);
layer.set_weights(weights, bias).unwrap();
let pixels: Vec<f32> = (1..=16).map(|v| v as f32).collect();
let x = Array::from_shape_vec((1, 4, 4, 1), pixels).unwrap().into_dyn();
let out = layer.forward(&x).unwrap();
// Valid padding: (4 - 2)/1 + 1 = 3 along each spatial axis -> [1, 3, 3, 1].
assert_eq!(out.shape(), &[1, 3, 3, 1]);
// An all-ones 2x2 kernel sums each window: the first is 1 + 2 + 5 + 6 = 14.
assert_eq!(out[[0, 0, 0, 0]], 14.0);
println!("output shape: {:?}", out.shape());
}
```
## 3.5.2. Constructors, Padding, and Output Shapes
The constructors take the hyperparameters as positional arguments. The kernel and stride argument shape tracks the rank. Every 1D layer uses a scalar `kernel_size` and `stride`. The 2D and 3D layers take tuples instead.
A separable layer inserts a `depth_multiplier` argument before the activation. A depthwise layer has no `filters` argument. Like Keras, it derives its output width from the input, and it takes its depth multiplier through a builder method.
```rust,ignore
Conv1D::new(filters, kernel_size: usize, input_shape: Vec<usize>, stride: usize, activation) -> Result<Conv1D, Error>
Conv2D::new(filters, kernel_size: (usize, usize), input_shape: Vec<usize>, strides: (usize, usize), activation) -> Result<Conv2D, Error>
Conv3D::new(filters, kernel_size: (usize, usize, usize), input_shape: Vec<usize>, strides: (usize, usize, usize), activation) -> Result<Conv3D, Error>
DepthwiseConv1D::new(kernel_size: usize, input_shape: Vec<usize>, stride: usize, activation) -> Result<DepthwiseConv1D, Error>
DepthwiseConv1D::with_depth_multiplier(self, depth_multiplier: usize) -> Result<DepthwiseConv1D, Error> // builder, defaults to 1
DepthwiseConv2D::new(kernel_size: (usize, usize), input_shape: Vec<usize>, strides: (usize, usize), activation) -> Result<DepthwiseConv2D, Error>
DepthwiseConv2D::with_depth_multiplier(self, depth_multiplier: usize) -> Result<DepthwiseConv2D, Error> // builder, defaults to 1
SeparableConv1D::new(filters, kernel_size: usize, input_shape: Vec<usize>, stride: usize, depth_multiplier: usize, activation) -> Result<SeparableConv1D, Error>
SeparableConv2D::new(filters, kernel_size: (usize, usize), input_shape: Vec<usize>, strides: (usize, usize), depth_multiplier: usize, activation) -> Result<SeparableConv2D, Error>
```
`activation` takes `impl Into<Activation>`. You can pass an `Activation` variant (`Activation::ReLU`, `Activation::Sigmoid`, `Activation::Tanh`, `Activation::Softmax`, `Activation::Linear`), or a standalone activation layer such as `ReLU::new()`. The `input_shape` you supply is the full expected input, including the batch dimension. Only `input_shape[1..]` (the channels and spatial extents) sizes the weights. The batch value you write is informational only. See [3.2. Dense Layers and Activations](./3.2._Dense_Layers_and_Activations.md) for more about the activation set.
3 builder methods refine a constructed layer. Each one consumes and returns `self`, so you can chain them. `with_padding(PaddingType)` switches the padding mode. `with_random_state(u64)` re-runs the Xavier initialization deterministically from a seed. Call it before you assign custom weights or start training (see [7.1. Reproducibility and Random Seeds](../Chapter-07/7.1._Reproducibility_and_Random_Seeds.md)).
`set_weights(...)` installs weights and a bias that you control. `set_weights` checks every array against the layer's expected shape. Note that a depthwise layer's `set_weights` takes a kernel and an `Array1` bias. A separable layer's takes 3 arrays instead: depthwise weights, pointwise weights, then bias.
Padding uses the `PaddingType` enum, which has 2 variants: `Valid` (the default) and `Same`. It sets the output size as follows:
- `Valid` applies no padding. It computes output values only where the kernel fully overlaps the input. The formula is `out = (in - k) / stride + 1` (integer floor division), on each spatial axis. Every input axis must be at least the kernel size, for the plain layers.
- `Same` zero-pads the borders so `out = ceil(in / stride)`. At `stride == 1` this keeps the spatial size exactly. At a larger stride, the output is the input length divided by the stride, rounded up. The layer splits the total padding evenly and puts the extra cell on the trailing edge (`pad_before = pad_total / 2`). This matches TensorFlow's `SAME` padding.
The following numbers show the rule in practice. A `[N, 8, 8, 1]` input through a `3x3` kernel gives 3 results. Valid padding at stride 1 gives `(8-3)/1+1 = 6`, so the output is `[N, 6, 6, F]`. Same padding at stride 1 gives `8`, so the output is `[N, 8, 8, F]`. Same padding at stride 2 gives `ceil(8/2) = 4`. A `Conv1D` over length 6, with kernel 3 and stride 2, under Valid padding, gives `(6-3)/2+1 = 2`.
`Conv3D` applies the same rule on depth, height, and width, each on its own. The 3 transposed layers take the same arguments and the same `PaddingType`. Both of their output rules grow the axis instead of shrinking it, and section 3.5.5 gives them. `model.summary()` prints these output shapes and the per-layer parameter counts. Use it to check a stack before you train it. See [3.1. Sequential Model](./3.1._The_Sequential_Model.md) for the full description of `summary`.
## 3.5.3. The im2col + GEMM Engine
`Conv1D`, `Conv2D`, and `Conv3D` do not each carry their own loop nest. A plain convolution is the same operation at every rank. Only the number of spatial axes changes. All 3 layers delegate their forward and backward numerics to 1 implementation in `convolution_engine.rs`.
This implementation is generic over the spatial rank `R = ndim - 2`. The layer wrapper keeps the public API, the weight storage, the activation, and the caches. The engine does the arithmetic.
The engine uses im2col plus GEMM, the same strategy the major frameworks use. For the forward pass, it gathers each output window into a row. This forms an `[out_plane, k_plane*Cin]` matrix, whose columns align with the flat weight matrix `[k_plane*Cin, F]`. A single matrix multiply then produces `[out_plane, F]`, and the engine adds the bias per filter.
Under the channels-last layout, this gather is a run of `Cin`-wide `copy_from_slice` calls, not a scalar-at-a-time walk. The product lands directly on a contiguous slab of the output, with no scatter. Trading a 6-deep loop nest for a matrix multiply lets the layer use the crate's tuned in-house GEMM, instead of a naive triple loop. See [6.2. Matrix Multiplication](../Chapter-06/6.2._Matrix_Multiplication.md) for more on that GEMM.
The backward pass runs 2 GEMMs per batch item. One computes the weight gradient. The other computes the input-gradient columns, which the engine then scatters back (col2im) into the input-gradient tensor.
The engine gates parallelism on estimated FLOPs, not on element counts. A `7x7x512` convolution and a `3x3x3` convolution can share the same output-element count, but their costs differ by a wide margin. The forward gate compares `2 * batch * F * out_plane * Cin*k` against `CONV_PARALLEL_MIN_FLOPS` (default 4,000,000, tunable at runtime through `rustyml::tuning`).
Below this threshold, the forward pass runs serial. Above it, the forward pass parallelizes over `(batch item, output-position block)` tasks. This lets a single large image fill every core, even at `batch == 1`. Each task builds its own im2col block and runs its own GEMM into a disjoint output region.
The backward pass parallelizes over batch items. It reduces the weight and bias partials in batch order, which keeps results bit-reproducible across runs on the same machine. It also routes each item's GEMMs through a switch. The switch keeps them parallel while the batch is too short to fill the thread pool. It flips them to serial once the batch alone saturates the pool. This way, a batch task never forks rayon again inside its own GEMM.
The parallel path and the serial path return the same numbers, so you rarely need to touch the gate. If you profile a workload that sits just under the threshold, see [7.3. Performance Tuning and Parallelism](../Chapter-07/7.3._Performance_Tuning_and_Parallelism.md) for how to move it.
## 3.5.4. Depthwise and Separable Convolutions
A standard convolution mixes across channels and across space in 1 step. Every output channel is a weighted sum over all input channels and all kernel taps. That coupling is where the parameters live. The weight tensor holds `F * Cin * kh * kw` values.
Depthwise separable convolution factors this operation into 2 cheaper stages. The first stage is a depthwise convolution that filters each input channel on its own (spatial mixing only, no cross-channel mixing). The second stage is a pointwise `1x1` convolution that recombines the channels (cross-channel mixing only, no spatial extent). This is the idea behind MobileNet and Xception. RustyML exposes both stages.
`DepthwiseConv2D` is the first stage alone. It carries `depth_multiplier` kernels of size `kh x kw` for each input channel, and emits `C * depth_multiplier` output channels. This is why it has no `filters` argument, exactly as in Keras. It does no channel recombination, so it cannot mix channels on its own. In practice, you almost always pair it with a `1x1` convolution downstream.
`depth_multiplier` defaults to 1. Set it with `with_depth_multiplier`, which returns a `Result` because it rejects 0. A separable layer takes its own `depth_multiplier` as a positional argument. It expands the intermediate channel count to `Cin * depth_multiplier`, before the pointwise stage collapses it back to `filters`.
The parameter counts are the point of this factorization. Consider `Cin` input channels, `F` output filters, and a `kh x kw` kernel:
- Standard `Conv2D`: `F * Cin * kh * kw + F`.
- `DepthwiseConv2D`: `C * dm * kh * kw + C * dm` (at the default `dm = 1`, this is `C * kh * kw + C`).
- `SeparableConv2D`: `dm * Cin * kh * kw` (depthwise) `+ F * Cin * dm` (pointwise) `+ F` (bias).
Ignoring the bias term, the separable-to-standard ratio is `1/F + 1/(kh*kw)`. The savings grow with both the filter count and the kernel area. Each parameter costs 1 multiply-accumulate per output position, so this same ratio is also the compute (FLOP) ratio.
As an example, take a 64-filter `3x3` convolution over 3 channels. The standard layer has `64*3*3*3 + 64 = 1792` parameters. The separable equivalent has `27 + 192 + 64 = 283` parameters, about 6.3 times fewer. A depthwise-only layer over those 3 channels has just `30` parameters. The following program builds all 3 layers and checks the counts.
```rust
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::Layer;
fn count(p: TrainingParameters) -> usize {
match p {
TrainingParameters::Trainable(n) | TrainingParameters::NonTrainable(n) => n,
TrainingParameters::NoTrainable => 0,
}
}
fn main() {
let input_shape = vec![1, 32, 32, 3]; // [batch, H, W, channels]
// Standard 64-filter 3x3 convolution over 3 input channels.
let standard =
Conv2D::new(64, (3, 3), input_shape.clone(), (1, 1), Activation::ReLU).unwrap();
// Separable equivalent: depthwise (depth_multiplier = 1) then a pointwise 1x1 to 64 filters.
let separable =
SeparableConv2D::new(64, (3, 3), input_shape.clone(), (1, 1), 1, Activation::ReLU).unwrap();
// Depthwise-only emits C * depth_multiplier = 3 channels and does no cross-channel mixing.
let depthwise = DepthwiseConv2D::new((3, 3), input_shape, (1, 1), Activation::ReLU).unwrap();
println!("standard Conv2D params: {}", count(standard.param_count()));
println!("separable Conv2D params: {}", count(separable.param_count()));
println!("depthwise Conv2D params: {}", count(depthwise.param_count()));
assert_eq!(count(standard.param_count()), 1792); // 64*3*3*3 + 64
assert_eq!(count(separable.param_count()), 283); // 27 + 192 + 64
assert_eq!(count(depthwise.param_count()), 30); // 3*3*3 + 3
}
```
The same math holds on a sequence, where wide channel counts are just as common. A 1D kernel drops the height axis, so a depthwise stage is `[k, C, dm]` and a pointwise stage is `[1, Cin*dm, F]`. Everything else carries over unchanged: the `c * dm + m` channel order, the padding rules, and the builder methods. The following program runs the same comparison over a 128-step sequence carrying 32 channels, where the separable form is about 4.5 times smaller.
```rust
use ndarray::Array3;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::Layer;
fn count(p: TrainingParameters) -> usize {
match p {
TrainingParameters::Trainable(n) | TrainingParameters::NonTrainable(n) => n,
TrainingParameters::NoTrainable => 0,
}
}
fn main() {
// [batch, length, channels]: 128 time steps carrying 32 channels.
let input_shape = vec![1, 128, 32];
// Standard 64-filter width-5 convolution over 32 input channels.
let standard = Conv1D::new(64, 5, input_shape.clone(), 1, Activation::ReLU).unwrap();
// Separable equivalent: depthwise width-5, then a pointwise 1-tap to 64 filters.
let mut separable =
SeparableConv1D::new(64, 5, input_shape.clone(), 1, 1, Activation::ReLU).unwrap();
// Depthwise-only emits C * depth_multiplier = 32 channels and mixes nothing across them.
let depthwise = DepthwiseConv1D::new(5, input_shape, 1, Activation::ReLU).unwrap();
println!("standard Conv1D params: {}", count(standard.param_count()));
println!("separable Conv1D params: {}", count(separable.param_count()));
println!("depthwise Conv1D params: {}", count(depthwise.param_count()));
assert_eq!(count(standard.param_count()), 10304); // 64*32*5 + 64
assert_eq!(count(separable.param_count()), 2272); // 160 + 2048 + 64
assert_eq!(count(depthwise.param_count()), 192); // 32*5 + 32
// Valid padding: (128 - 5)/1 + 1 = 124 positions, each carrying 64 filters.
let x = Array3::<f32>::zeros((1, 128, 32)).into_dyn();
assert_eq!(separable.forward(&x).unwrap().shape(), &[1, 124, 64]);
}
```
Use these layers when channel counts run high and you want to cut both parameters and compute. Wide feature maps are one example. A model meant to run on modest hardware is another. In exchange, you accept a modeling tradeoff: the factored form is strictly less expressive than a full convolution, for the same `F` and kernel.
Their engine differs from the standard layers' engine. A depthwise pass uses a direct loop nest, because the channels are independent and im2col buys little here. It parallelizes over `(batch item, output row)` tasks. Output rows are disjoint, so this split needs no merge. It gates on `NAIVE_CONV_PARALLEL_MIN_FLOPS` (default 1,000,000).
A separable layer runs its depthwise stage through this same naive path. It then routes its pointwise stage back through the shared im2col plus GEMM engine. A 1-tap convolution is exactly a per-position cross-channel matrix multiply.
All 4 layers share 1 depthwise loop nest, across both ranks. A `[batch, length, channels]` tensor holds its values in the same order as `[batch, 1, length, channels]`, and a `[k, C, dm]` kernel in the same order as `[1, k, C, dm]`. The 1D layers therefore set the height terms of the shared geometry to 1 and pass the flat slices of their own rank-3 arrays. This repacks nothing, and the 1D and the 2D form cannot drift apart.
## 3.5.5. Transposed Convolutions
`Conv1DTranspose`, `Conv2DTranspose`, and `Conv3DTranspose` run a convolution backwards over the spatial axes. A convolution shrinks a tensor, so its transpose grows one. This is the layer a decoder or a generator uses to climb back to the resolution the matching `Conv2D` consumed. Unlike `UpSampling2D`, it learns how it grows, because it carries a kernel and a bias.
The name is literal. A convolution is a linear map, so it has a transpose, and that transpose is what this layer computes. The forward pass here is exactly the gradient a plain convolution computes with respect to its input. The backward pass here is exactly a plain forward pass. RustyML implements it that way. The transposed engine reuses every geometry helper of the standard engine and only reverses the direction of its 2 matrix products.
So a transposed convolution costs about what the convolution it transposes costs. Read "deconvolution" as a synonym to avoid. The layer recovers the shape, never the values.
Each input position writes its own value times the whole kernel into the output, starting at `position * stride`. Wherever the stride is below the kernel size, neighboring windows overlap, and the overlapping writes accumulate. Wherever the stride is above the kernel size, some output positions receive nothing at all, and those hold exactly the bias. The layer adds the bias once per output position, at the end.
The output size follows 1 rule per padding mode, applied to each axis on its own:
| Padding | Output size of 1 axis | At stride 1 |
| --- | --- | --- |
| `Valid` | `input * stride + max(kernel - stride, 0)` | `input + kernel - 1` |
| `Same` | `input * stride` | `input` |
The constructors mirror the plain ones argument for argument, and the 2 builder methods `with_padding` and `with_random_state` refine them:
```rust,ignore
Conv1DTranspose::new(filters, kernel_size: usize, input_shape: Vec<usize>, stride: usize, activation) -> Result<Conv1DTranspose, Error>
Conv2DTranspose::new(filters, kernel_size: (usize, usize), input_shape: Vec<usize>, strides: (usize, usize), activation) -> Result<Conv2DTranspose, Error>
Conv3DTranspose::new(filters, kernel_size: (usize, usize, usize), input_shape: Vec<usize>, strides: (usize, usize, usize), activation) -> Result<Conv3DTranspose, Error>
```
1 difference in the weight layout is easy to miss. The kernel is `[k..., filters, channels]`, so the filter axis comes **before** the input-channel axis. That is the reverse of the plain convolution kernel. It is also the same order Keras uses for these layers. The reason is the direction of the pass: a transposed convolution reads `channels` and writes `filters`.
`set_weights` therefore refuses a kernel laid out for the matching `Conv2D` when the filter count and the channel count differ. When the 2 counts are equal, the 2 layouts have the same shape, and no shape check can separate them. `param_count` is unaffected, since the product is the same either way.
A second difference is that these layers put no lower bound on the input spatial size. `Conv2D::new` rejects an input smaller than its kernel, because such a convolution has no valid window. A transposed convolution has the opposite problem, so a `1x1` input under a `3x3` kernel is legal. This is the normal first step of a decoder that starts from a `Dense` head.
The following example grows a `2x2` feature map into `4x4` and checks 2 of the 16 output values by hand.
```rust
use ndarray::{Array, Array1, Array4};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::Layer;
fn main() {
// Channels-last: [batch, height, width, channels].
let mut layer =
Conv2DTranspose::new(1, (3, 3), vec![1, 2, 2, 1], (2, 2), Activation::Linear)
.unwrap()
.with_padding(PaddingType::Same);
// Weights are [kernel_h, kernel_w, filters, channels]: the filter axis comes first.
let taps: Vec<f32> = (1..=9).map(|v| v as f32).collect();
let weights = Array4::from_shape_vec((3, 3, 1, 1), taps).unwrap();
layer.set_weights(weights, Array1::zeros(1)).unwrap();
let x = Array::from_shape_vec((1, 2, 2, 1), vec![1.0f32, 2.0, 3.0, 4.0])
.unwrap()
.into_dyn();
let out = layer.forward(&x).unwrap();
// Same padding: 2 * 2 = 4 along each spatial axis -> [1, 4, 4, 1].
assert_eq!(out.shape(), &[1, 4, 4, 1]);
// Input (0, 0) scales the whole kernel from output (0, 0), so out[0, 0] is 1 * w[0, 0].
assert_eq!(out[[0, 0, 0, 0]], 1.0);
// Output (2, 2) is the one position all 4 inputs reach: 1*9 + 2*7 + 3*3 + 4*1 = 36.
assert_eq!(out[[0, 2, 2, 0]], 36.0);
println!("output shape: {:?}", out.shape());
}
```
The overlap has a visible cost. When the stride does not divide the kernel size, some output positions collect more kernel taps than their neighbors do. A trained model turns that imbalance into a regular grid of bright and dark cells. This is the checkerboard artifact.
A kernel size that the stride divides evenly removes it. A `4x4` kernel at stride 2 is a safer default than a `3x3` kernel at stride 2. The alternative is `UpSampling2D` followed by a plain `Conv2D`, which cannot produce the artifact at all.
The round trip back to the original size is exact only when the convolution kept every input position. Under `Valid` that means the kernel is at least the stride, and the stride divides `input - kernel`. The convolution otherwise drops the trailing positions its last window could not reach, and the transposed convolution has nothing to rebuild them from. An 8-wide axis at kernel 3 and stride 2 convolves to 3 positions, and those 3 positions transpose back to 7, not 8.
Under `Same` the condition is that the stride divides the input, and a stride that does not divide it overshoots to the next multiple instead. Keras covers this with an `output_padding` argument. RustyML has no such argument. Pick a stride that divides, or resize afterward with a border layer from section 3.5.7.
The 3 layers share the standard engine's parallelism gate, `tuning::conv::set_parallel_min_flops`, because both engines run the same 2 matrix-product shapes. The transposed forward pass parallelizes over batch items only. Its scatter accumulates into overlapping output positions, so splitting 1 image across threads would need a merge that batch items never need. Below a full batch the per-item products run in parallel instead, so a batch of 1 still uses every core.
## 3.5.6. Building a Small CNN
Convolutions emit rank-3, rank-4, or rank-5 tensors, but a `Dense` classifier head needs a rank-2 `[batch, features]` matrix. `Flatten` bridges the two. `Flatten::new(input_shape: Vec<usize>)` builds a parameter-free layer that reshapes `[batch, ...]` into `[batch, product-of-the-rest]`. It accepts 3D, 4D, or 5D input at forward time. You give it the shape of the tensor entering it, which is the convolution's output shape. It then works out the flattened feature count.
`Reshape` is the general form of the same idea. `Reshape::new(target_shape: Vec<isize>)` builds a parameter-free layer that rewrites every axis after the batch axis. The `target_shape` never names the batch axis. Axis 0 passes through untouched, so 1 instance serves every batch size. `Flatten::new` differs here, because it takes a fixed input shape and binds to it at construction.
At most 1 entry of `target_shape` may be `-1`. That axis takes whatever extent makes the element count match. `Reshape::new(vec![-1, 2])` on a `[batch, 4]` input gives `[batch, 2, 2]`, since `4 / 2 = 2`. `Reshape::new(vec![-1])` collapses every axis after the batch axis into 1 axis, so it is exactly `Flatten`. An empty `target_shape` is legal, and gives the rank-1 output shape `[batch]`.
The 2 layers serve different directions. `Flatten` goes from a convolution stack into a `Dense` head. `Reshape` also goes the other way, and turns a `Dense` output back into a volume. A decoder needs that direction.
From there, the transposed convolutions of section 3.5.5 grow the spatial axes back with a learned kernel. The parameter-free `UpSampling1D`, `UpSampling2D`, and `UpSampling3D` layers do the same without one. See [3.6. Pooling Layers](./3.6._Pooling_Layers.md) for the upsampling shapes and their `Interpolation` modes.
The constructor rejects more than 1 `-1`, a `0`, and any value below `-1`, and each case returns `Error::InvalidParameter`. An element count that cannot match is a forward-time error, and returns `Error::ShapeMismatch`. For example, `vec![2, 3]` needs 6 elements per sample, and a `[5, 4]` input holds only 4.
`Reshape::new` rejects a `0` in `target_shape` on purpose. No non-empty input can ever match a `0` axis. The constructor raises this error early, not at the first forward pass. The set of valid programs stays the same, and only the moment of the error moves earlier.
Like `Flatten`, `Reshape` moves no data. It reads and writes in C order, so the last axis varies fastest. The channels-last layout puts the channel axis innermost, so a reshape that splits or merges the trailing axes regroups channels before spatial positions. `Flatten` uses that same order, so the 2 layers agree on where each element lands. Read a target shape with that order in mind.
The following program folds a rank-2 batch into a volume through 1 inferred axis:
```rust
use ndarray::Array2;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::Layer;
fn main() {
// 5 samples of 4 features each.
let x = Array2::<f32>::from_shape_fn((5, 4), |(n, k)| (n * 4 + k) as f32).into_dyn();
// The target shape names the axes after the batch axis. -1 takes 4 / 2 = 2.
let mut layer = Reshape::new(vec![-1, 2]).unwrap();
let folded = layer.forward(&x).unwrap();
assert_eq!(folded.shape(), &[5, 2, 2]);
// C order: the last axis varies fastest, so sample 0 folds into [[0, 1], [2, 3]].
assert_eq!(folded[[0, 1, 0]], 2.0);
// A gradient of the output shape restores the input shape.
let grad = layer.backward(&folded).unwrap();
assert_eq!(grad.shape(), &[5, 4]);
println!("folded shape: {:?}", folded.shape());
}
```
The next example builds a complete stack. A `Conv2D` runs over synthetic single-channel `8x8` images. `Flatten` then feeds the result into a `Dense` regression head. [The Sequential model](./3.1._The_Sequential_Model.md) trains the stack for a few epochs.
```rust
use ndarray::{Array2, Array4};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::losses::*;
use rustyml::neural_network::optimizers::*;
use rustyml::neural_network::sequential::Sequential;
fn main() {
// Synthetic "images": 6 samples, 8x8, 1 channel.
let mut x = Array4::<f32>::zeros((6, 8, 8, 1));
for n in 0..6 {
for i in 0..8 {
for j in 0..8 {
x[[n, i, j, 0]] = ((n + i + j) as f32 * 0.1).sin();
}
}
}
let x = x.into_dyn();
// 3 regression targets per sample.
let y = Array2::<f32>::from_shape_fn((6, 3), |(n, k)| (n as f32) * 0.01 + (k as f32) * 0.1)
.into_dyn();
let mut model = Sequential::new();
model
// [6, 8, 8, 1] -> Conv2D(4 filters, 3x3, Valid) -> [6, 6, 6, 4]
.add(
Conv2D::new(4, (3, 3), vec![6, 8, 8, 1], (1, 1), Activation::ReLU)
.unwrap()
.with_random_state(42),
)
// [6, 6, 6, 4] -> Flatten -> [6, 144]
.add(Flatten::new(vec![6, 6, 6, 4]).unwrap())
// [6, 144] -> Dense -> [6, 3]
.add(Dense::new(6 * 6 * 4, 3, Activation::Linear).unwrap().with_random_state(7))
.compile(SGD::new(0.01, 0.0, false, 0.0).unwrap(), MeanSquaredError::new());
model.summary();
model.fit(&x, &y, 3).unwrap();
let pred = model.predict(&x).unwrap();
assert_eq!(pred.shape(), &[6, 3]);
println!("prediction shape: {:?}", pred.shape());
}
```
The `Dense` input dimension is not a guess. It is the flattened feature count `6 * 6 * 4 = 144`, which you can read directly from the convolution's output shape. Get it wrong, and the `Dense` layer's matrix multiply rejects the batch. `summary()` prints the shape at each stage and the parameter budget. This is the fastest way to catch a mis-sized head:
```text
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Layer (type) ┃ Output Shape ┃ Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ conv2d (Conv2D) │ (6, 6, 6, 4) │ 40 │
│ flatten (Flatten) │ (None, 144) │ 0 │
│ dense (Dense) │ (None, 3) │ 435 │
└─────────────────────────────────┴────────────────────────┴───────────────┘
Total params: 475 (1900 B)
Trainable params: 475 (1900 B)
Non-trainable params: 0 (0 B)
```
Insert a [pooling layer](./3.6._Pooling_Layers.md) between the convolution and the flatten step, if you want to shrink the spatial extent before the dense head. Save the trained stack with the tools in [3.9. Saving and Loading Weights](./3.9._Saving_and_Loading_Weights.md).
## 3.5.7. Explicit Borders: ZeroPadding and Cropping
`PaddingType::Same` is a policy. It computes the total padding for you. It splits an odd total by a fixed rule, `pad_before = pad_total / 2`, so the extra cell lands on the trailing edge. When you want a different split, or padding that no convolution follows, reach for a border layer instead.
The family has 6 members in 2 halves. `ZeroPadding1D`, `ZeroPadding2D`, and `ZeroPadding3D` add zero positions at the ends of the spatial axes. `Cropping1D`, `Cropping2D`, and `Cropping3D` remove positions there. Each half is the backward pass of the other half. All 6 leave the batch axis and the channel axis untouched, and none of them holds a parameter.
| Layer | Input tensor | Output tensor |
| --- | --- | --- |
| `ZeroPadding1D` | `[N, L, C]` | `[N, L + before + after, C]` |
| `ZeroPadding2D` | `[N, H, W, C]` | `[N, H + top + bottom, W + left + right, C]` |
| `ZeroPadding3D` | `[N, D, H, W, C]` | each spatial axis grows by its 2 amounts |
| `Cropping1D` | `[N, L, C]` | `[N, L - before - after, C]` |
| `Cropping2D` | `[N, H, W, C]` | `[N, H - top - bottom, W - left - right, C]` |
| `Cropping3D` | `[N, D, H, W, C]` | each spatial axis shrinks by its 2 amounts |
Every constructor takes 1 argument and returns the layer directly, with no `Result`. A border amount is a `usize`, so no value is invalid at construction. A cropping layer that removes too much fails at forward time instead, when the input extent is known.
The argument accepts 3 forms, and the rank decides which ones apply:
| Layer rank | Integer `n` | Tuple | Tuple of pairs |
| --- | --- | --- | --- |
| 1D | `n` at both ends | `(before, after)` | not applicable |
| 2D | `n` at all 4 edges | `(height, width)`, equal at both ends of each axis | `((top, bottom), (left, right))` |
| 3D | `n` at all 6 faces | `(dim1, dim2, dim3)`, equal at both ends of each axis | 3 `(before, after)` pairs |
Read the 2D tuple form with care. `ZeroPadding2D::new((1, 2))` gives 1 row at the top *and* the bottom, plus 2 columns at the left *and* the right. It does not give 1 row at the top and 2 at the bottom. Only the 1D pair form names the 2 ends of a single axis.
A cropping layer must leave at least 1 position on each spatial axis. `Cropping1D::new((2, 3))` on a length-5 input removes all 5 steps, so `forward` returns `Error::InvalidInput`. `Cropping1D::new((2, 2))` on the same input leaves 1 step, and it succeeds.
Neither half moves data across the channel axis. A pad allocates a zero tensor and copies the input into the middle of it. A crop copies the interior out. Because each half is the other half's backward pass, a `ZeroPadding2D` followed by a `Cropping2D` with the same amounts is the identity on values.
The following program shows the round trip, then puts a pad in front of a Valid convolution so the convolution keeps the spatial size:
```rust
use ndarray::{Array2, Array4};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::losses::*;
use rustyml::neural_network::optimizers::*;
use rustyml::neural_network::sequential::Sequential;
use rustyml::neural_network::traits::Layer;
fn main() {
// 4 samples, 6x6 pixels, 1 channel. Every value is above 0.
let x = Array4::<f32>::from_shape_fn((4, 6, 6, 1), |(n, i, j, _)| {
((n + i + j) as f32 + 1.0) * 0.05
})
.into_dyn();
// 1 zero row at the top only, and 1 zero column at each side: 6x6 -> 7x8.
let mut pad = ZeroPadding2D::new(((1, 0), (1, 1)));
let padded = pad.forward(&x).unwrap();
assert_eq!(padded.shape(), &[4, 7, 8, 1]);
assert_eq!(padded[[0, 0, 0, 0]], 0.0);
// A crop with the same amounts cancels the pad exactly.
let mut crop = Cropping2D::new(((1, 0), (1, 1)));
assert_eq!(crop.forward(&padded).unwrap(), x);
// In a model: 6x6 grows to 8x8, so the 3x3 Valid convolution gives 6x6 back.
let y = Array2::<f32>::from_shape_fn((4, 2), |(n, k)| (n as f32) * 0.1 + (k as f32) * 0.01)
.into_dyn();
let mut model = Sequential::new();
model
.add(ZeroPadding2D::new(1))
.add(Conv2D::new(2, (3, 3), vec![4, 8, 8, 1], (1, 1), Activation::ReLU).unwrap())
.add(Flatten::new(vec![4, 6, 6, 2]).unwrap())
.add(Dense::new(6 * 6 * 2, 2, Activation::Linear).unwrap())
.compile(SGD::new(0.01, 0.0, false, 0.0).unwrap(), MeanSquaredError::new());
model.fit(&x, &y, 3).unwrap();
let pred = model.predict(&x).unwrap();
assert_eq!(pred.shape(), &[4, 2]);
println!("prediction shape: {:?}", pred.shape());
}
```
`summary()` prints `Unknown` for a border layer until the first forward pass. A border layer takes no `input_shape` argument, so it derives its output shape from the tensor it receives. It cannot answer earlier. `Flatten` and the pooling layers instead bind to a declared `input_shape` at construction, and they print their output shape right away.
## 3.5.8. Permute and RepeatVector
2 more parameter-free layers round out the shape family. `Permute` reorders the axes after the batch axis. `RepeatVector` turns 1 vector per sample into a sequence of identical steps.
`Permute::new(dims)` names the new order. `dims` counts from 1 and never includes the batch axis. `Permute::new(vec![2, 1])` on a `[batch, steps, features]` input gives `[batch, features, steps]`. The entries must be a permutation of `1..=dims.len()`, so each axis appears exactly once. The input rank must be `dims.len() + 1`, and a rank the layer does not serve is a forward-time `Error::InvalidInput`.
`Permute::new(vec![])` returns `Error::InvalidParameter`. Such a layer reorders nothing, and every other layer here needs a batch axis and at least 1 more axis.
`Permute` differs from `Reshape` in a way worth stating plainly. A reshape reads the same buffer in the same order under a new shape, so every value keeps its place in memory order. A permute reads the buffer in a new order, so the values land at new positions. Both layers allocate a new tensor, but only the permute pays for a scattered copy.
A permute that leaves the last axis last keeps whole rows contiguous and stays close to copy speed. A permute that moves the last axis cuts the contiguous run to 1 element and costs several times more. When the model allows a choice, put the permute where the tensor is small.
`RepeatVector::new(n)` takes a `[batch, features]` input and gives `[batch, n, features]`. Every one of the `n` steps holds the same vector. `n` must be greater than 0, and a `0` is an `Error::InvalidParameter` at construction. The input rank must be 2.
`RepeatVector` exists for the decoder side of an encoder-decoder model. A recurrent layer here returns only its last hidden state, a rank-2 tensor, and a recurrent layer needs a rank-3 input. `RepeatVector` bridges the 2, so an `LSTM` can feed another `LSTM`. This layer emits the same vector at every step, which is the standard way to seed a decoder with a fixed context. See [3.7. Recurrent Layers](./3.7._Recurrent_Layers.md) for what a recurrent layer returns, and why the bridge is needed.
The following program runs both layers by hand:
```rust
use ndarray::{Array2, Array3};
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::Layer;
fn main() {
// 2 samples, 2 steps, 3 features.
let x = Array3::<f32>::from_shape_fn((2, 2, 3), |(n, t, f)| (n * 6 + t * 3 + f) as f32)
.into_dyn();
// dims counts from 1 and skips the batch axis, so (2, 1) swaps steps and features.
let mut permute = Permute::new(vec![2, 1]).unwrap();
let swapped = permute.forward(&x).unwrap();
assert_eq!(swapped.shape(), &[2, 3, 2]);
// Sample 0 was [[0, 1, 2], [3, 4, 5]], and it transposes to [[0, 3], [1, 4], [2, 5]].
assert_eq!(swapped[[0, 1, 0]], 1.0);
assert_eq!(swapped[[0, 0, 1]], 3.0);
// The gradient runs back through the inverse order.
let grad = permute.backward(&swapped).unwrap();
assert_eq!(grad.shape(), &[2, 2, 3]);
// RepeatVector turns 1 vector per sample into a sequence of identical steps.
let state = Array2::<f32>::from_shape_fn((2, 3), |(n, f)| (n * 3 + f) as f32).into_dyn();
let mut repeat = RepeatVector::new(4).unwrap();
let sequence = repeat.forward(&state).unwrap();
assert_eq!(sequence.shape(), &[2, 4, 3]);
assert_eq!(sequence[[0, 0, 2]], sequence[[0, 3, 2]]);
println!("swapped {:?}, sequence {:?}", swapped.shape(), sequence.shape());
}
```
Both layers report `Unknown` from `summary()` until the first forward pass, for the same reason the border layers do. Neither one takes an input shape, so neither can name its output shape earlier.
## 3.5.9. Identity
`Identity` is the layer that does nothing. It returns its input unchanged at every rank and every shape, and its backward pass returns the gradient it receives. `Identity::new()` takes no argument and returns the layer directly, with no `Result`, because it has nothing to misconfigure. It also implements `Default`.
A layer that does nothing sounds useless until a program builds a model instead of a person writing it out by hand. A function that picks among several layers needs something to return when the choice is "no operation". A stack whose depth is a runtime value needs a filler. An experiment that compares an architecture with and without a layer needs both models to keep the same layer count. This way, `summary()` and a saved weight file still line up. All 3 read better with a layer that does nothing than with an `Option` at every position.
The layer copies. It cannot borrow, because a layer returns an owned tensor. The copy is 1 linear pass and runs at memory speed, but it is not free. Remove the layer rather than keep it in a model that is finished.
A rank-0 tensor has no batch axis and gives `Error::InvalidInput`. A tensor with a zero extent gives `Error::EmptyInput`. `Reshape`, `Permute`, `RepeatVector`, and the border layers draw the same 2 boundaries.
```rust
use ndarray::Array2;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::losses::*;
use rustyml::neural_network::optimizers::*;
use rustyml::neural_network::sequential::Sequential;
use rustyml::neural_network::traits::Layer;
fn main() {
// 2 samples, 3 features.
let x = Array2::<f32>::from_shape_fn((2, 3), |(n, f)| (n * 3 + f) as f32).into_dyn();
let mut identity = Identity::new();
let out = identity.forward(&x).unwrap();
assert_eq!(out, x);
// The gradient goes back exactly as it arrived.
let grad = identity.backward(&out).unwrap();
assert_eq!(grad, out);
// A stack whose depth is a runtime value. The filler keeps the layer count fixed.
let depth = 2;
let mut model = Sequential::new();
for step in 0..3 {
if step < depth {
model.add(Dense::new(3, 3, Linear::new()).unwrap());
} else {
model.add(Identity::new());
}
}
model.compile(
SGD::new(0.01, 0.0, false, 0.0).unwrap(),
MeanSquaredError::new(),
);
model.summary();
println!("output shape {:?}", model.predict(&x).unwrap().shape());
}
```
`summary()` prints `Unknown` for this layer until the first forward pass, for the same reason the border layers do. It takes no input shape, so it cannot name its output shape earlier.
## 3.5.10. Errors and How to Avoid Them
The convolution layers and the shape layers validate their configuration closely. They return typed errors instead of a panic on bad input. The table below lists the common cases and their error variants.
| Condition | Error |
| --- | --- |
| `filters` is `0`, for `Conv1D`, `Conv2D`, `Conv3D`, any of the 3 transposed layers, or `SeparableConv2D` (`DepthwiseConv2D` has no `filters` argument) | `Error::InvalidParameter` |
| A kernel dim or a stride is `0` | `Error::InvalidParameter` |
| `depth_multiplier == 0` (`SeparableConv2D::new`, `DepthwiseConv2D::with_depth_multiplier`) | `Error::InvalidParameter` |
| `input_shape` wrong rank, zero channels, or (plain convolutions only) smaller than the kernel (constructor) | `Error::InvalidInput` |
| `forward` handed a tensor of the wrong rank | `Error::InvalidInput` |
| Plain Valid convolution whose runtime spatial dim is below the kernel | `Error::InvalidInput` |
| `DepthwiseConv2D` runtime channel count differs from the declared channel count | `Error::DimensionMismatch` |
| A transposed convolution handed a runtime channel count its kernel was not built for, or a tensor with a 0-length spatial axis | `Error::InvalidInput` |
| A transposed convolution's `backward` handed a gradient that is not the forward output shape | `Error::ShapeMismatch` |
| A border layer handed a tensor of the wrong rank, or one with a zero extent | `Error::InvalidInput` or `Error::EmptyInput` |
| A `Cropping*` amount that leaves a spatial axis with no positions | `Error::InvalidInput` |
| `Permute::new` given a `dims` that is empty or is not a permutation of `1..=dims.len()` | `Error::InvalidParameter` |
| `RepeatVector::new(0)` | `Error::InvalidParameter` |
| `Identity` handed a rank-0 tensor, or one with a zero extent | `Error::InvalidInput` or `Error::EmptyInput` |
| `set_weights` shape does not match | `Error::NeuralNetwork(NnError::WeightShape)` |
| `backward` called before `forward` | `Error::NeuralNetwork(NnError::ForwardPassNotRun)` |
2 of these error paths need more explanation. The engine catches a kernel larger than the input, under Valid padding, at 2 points. It checks at construction, against the declared `input_shape`. It checks again at forward time, against the actual tensor. The engine computes `in - k` in `usize`. Rather than let that computation underflow, it returns `InvalidInput`.
A channel mismatch is a recoverable error only on `DepthwiseConv2D`. This layer checks the runtime channel count directly, and returns `DimensionMismatch` when the count does not match. The standard `Conv1D`, `Conv2D`, and `Conv3D` layers size their weight matrix from the channel count in the declared `input_shape`. They do not check this count again on every forward call.
The 3 transposed layers do check it, and return `InvalidInput` on a mismatch, because their pass reads the channel count out of the kernel. You must always supply the channel count you declared. Treat the declared channel count as a contract.
The following program exercises these recoverable paths:
```rust
use ndarray::Array;
use rustyml::error::Error;
use rustyml::neural_network::layers::*;
use rustyml::neural_network::traits::Layer;
fn main() {
// 1. Kernel larger than the declared input under Valid padding: rejected at construction.
let too_big = Conv2D::new(1, (3, 3), vec![1, 2, 2, 1], (1, 1), Activation::Linear);
assert!(matches!(too_big, Err(Error::InvalidInput(_))));
// 2. A zero depth multiplier is rejected by the builder rather than panicking.
let bad_dm = DepthwiseConv2D::new((2, 2), vec![1, 4, 4, 2], (1, 1), Activation::Linear)
.unwrap()
.with_depth_multiplier(0);
assert!(matches!(bad_dm, Err(Error::InvalidParameter { .. })));
// 3. A runtime tensor smaller than the kernel: Valid geometry returns an error, not a panic.
let mut conv = Conv2D::new(1, (3, 3), vec![1, 5, 5, 1], (1, 1), Activation::Linear).unwrap();
let small = Array::ones((1, 2, 5, 1)).into_dyn(); // height 2 < kernel 3
assert!(matches!(conv.forward(&small), Err(Error::InvalidInput(_))));
// 4. DepthwiseConv2D turns a runtime channel mismatch into a recoverable error.
let mut dw =
DepthwiseConv2D::new((2, 2), vec![1, 4, 4, 2], (1, 1), Activation::Linear).unwrap();
let wrong_channels = Array::ones((1, 4, 4, 3)).into_dyn(); // 3 channels, layer expects 2
assert!(matches!(
dw.forward(&wrong_channels),
Err(Error::DimensionMismatch { .. })
));
println!("all error paths behaved as documented");
}
```
The `ForwardPassNotRun` error most often surprises you during training. The backward pass reads caches that `forward` writes. Call `backward` on a fresh layer, or twice in a row without a `forward` call between them. Either way, the layer returns this variant instead of reading stale state.
Inside `Sequential`, the framework handles this order for you. You meet this error only when you drive layers by hand. See [1.6. Error Handling](../Chapter-01/1.6._Error_Handling.md) for the full error taxonomy and the smart constructors behind these variants.