Struct dfdx::nn::BatchNorm2D
source · [−]pub struct BatchNorm2D<const C: usize> {
pub scale: Tensor1D<C>,
pub bias: Tensor1D<C>,
pub running_mean: Tensor1D<C>,
pub running_var: Tensor1D<C>,
pub epsilon: f32,
pub momentum: f32,
}
Expand description
Batch normalization for images as described in Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Generics:
C
the size of the spatial dimension to reduce. For 3d tensors this is the 0th dimension. For 4d tensors, this is the 1st dimension.
Training vs Inference
BatchNorm2D supports the following cases (see sections below for more details):
- Training: ModuleMut and OwnedTape on the input tensor
- Inference: Module and NoneTape on the input tensor.
NOTE: ModuleMut/NoneTape, and Module/OwnedTape will fail to compile.
Examples:
let bn: BatchNorm2D<3> = Default::default();
let _ = bn.forward(Tensor3D::<3, 2, 2>::zeros());
let _ = bn.forward(Tensor4D::<4, 3, 2, 2>::zeros());
Training
- Running statistics: updated with momentum
- Normalization: calculated using batch stats
Inference
- Running statistics: not updated
- Normalization: calculated using running stats
Fields
scale: Tensor1D<C>
Scale for affine transform. Defaults to 1.0
bias: Tensor1D<C>
Bias for affine transform. Defaults to 0.0
running_mean: Tensor1D<C>
Spatial mean that is updated during training. Defaults to 0.0
running_var: Tensor1D<C>
Spatial variance that is updated during training. Defaults to 1.0
epsilon: f32
Added to variance before taking sqrt for numerical stability. Defaults to 1e-5
momentum: f32
Controls exponential moving average of running stats.Defaults to 0.1
running_stat * (1.0 - momentum) + stat * momentum
.
Trait Implementations
sourceimpl<const C: usize> CanUpdateWithGradients for BatchNorm2D<C>
impl<const C: usize> CanUpdateWithGradients for BatchNorm2D<C>
sourcefn update<G: GradientProvider>(
&mut self,
grads: &mut G,
unused: &mut UnusedTensors
)
fn update<G: GradientProvider>(
&mut self,
grads: &mut G,
unused: &mut UnusedTensors
)
G
, then this function should
add the tensor’s UniqueId to UnusedTensors. Read moresourceimpl<const C: usize> Clone for BatchNorm2D<C>
impl<const C: usize> Clone for BatchNorm2D<C>
sourcefn clone(&self) -> BatchNorm2D<C>
fn clone(&self) -> BatchNorm2D<C>
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moresourceimpl<const C: usize> Debug for BatchNorm2D<C>
impl<const C: usize> Debug for BatchNorm2D<C>
sourceimpl<const C: usize> Default for BatchNorm2D<C>
impl<const C: usize> Default for BatchNorm2D<C>
sourceimpl<const C: usize> LoadFromNpz for BatchNorm2D<C>
impl<const C: usize> LoadFromNpz for BatchNorm2D<C>
sourceimpl<const C: usize, const H: usize, const W: usize> Module<Tensor3D<C, H, W, NoneTape>> for BatchNorm2D<C>
impl<const C: usize, const H: usize, const W: usize> Module<Tensor3D<C, H, W, NoneTape>> for BatchNorm2D<C>
sourcefn forward(&self, x: Tensor3D<C, H, W, NoneTape>) -> Self::Output
fn forward(&self, x: Tensor3D<C, H, W, NoneTape>) -> Self::Output
Inference 3d forward - does not update Self::running_mean and Self::running_var
sourceimpl<const B: usize, const C: usize, const H: usize, const W: usize> Module<Tensor4D<B, C, H, W, NoneTape>> for BatchNorm2D<C>
impl<const B: usize, const C: usize, const H: usize, const W: usize> Module<Tensor4D<B, C, H, W, NoneTape>> for BatchNorm2D<C>
sourcefn forward(&self, x: Tensor4D<B, C, H, W, NoneTape>) -> Self::Output
fn forward(&self, x: Tensor4D<B, C, H, W, NoneTape>) -> Self::Output
Inference 4d forward - does not update Self::running_mean and Self::running_var
sourceimpl<const C: usize, const H: usize, const W: usize> ModuleMut<Tensor3D<C, H, W, OwnedTape>> for BatchNorm2D<C>
impl<const C: usize, const H: usize, const W: usize> ModuleMut<Tensor3D<C, H, W, OwnedTape>> for BatchNorm2D<C>
sourcefn forward_mut(&mut self, x: Tensor3D<C, H, W, OwnedTape>) -> Self::Output
fn forward_mut(&mut self, x: Tensor3D<C, H, W, OwnedTape>) -> Self::Output
Training 3d forward - updates Self::running_mean and Self::running_var
sourceimpl<const B: usize, const C: usize, const H: usize, const W: usize> ModuleMut<Tensor4D<B, C, H, W, OwnedTape>> for BatchNorm2D<C>
impl<const B: usize, const C: usize, const H: usize, const W: usize> ModuleMut<Tensor4D<B, C, H, W, OwnedTape>> for BatchNorm2D<C>
sourcefn forward_mut(&mut self, x: Tensor4D<B, C, H, W, OwnedTape>) -> Self::Output
fn forward_mut(&mut self, x: Tensor4D<B, C, H, W, OwnedTape>) -> Self::Output
Training 4d forward - updates Self::running_mean and Self::running_var