rustyml 0.17.0

//! Neural network primitives: layers, loss functions, optimizers, the 2 models, and the traits
//! that tie them together
//!
//! The module builds, trains, and runs neural networks.
//! Layers, optimizers, and losses live in the [`layers`], [`optimizers`], and [`losses`]
//! submodules. [`Sequential`] stacks layers into a chain, and [`Graph`] wires them into a directed
//! graph. The shared interfaces live in [`traits`]. Every tensor in the framework is a [`Tensor`],
//! an `f32` n-dimensional array
//!
//! A layer computes, and it holds no cache and no gradient.
//! [`forward`] and [`backward`] take `&self` and a [`Ctx`]. The context carries the training flag,
//! the caches of the pass, every parameter gradient, and the non-trainable state that a training
//! pass changes. [`Ctx::training`] and [`Ctx::inference`] pick the mode. A built model is therefore
//! `Send` and `Sync`, and several threads can run inference against 1 model
//!
//! A layer can hold other layers, which are its sublayers.
//! [`LayerBase::sublayers`] lists them by name, and the layer calls each sublayer inside
//! [`Ctx::sublayer`]. A [`LayerPath`] addresses each layer of that tree. The path `2.forward` is
//! the sublayer `forward` of the layer at model position 2.
//!
//! The gradient store, the optimizer
//! state, the state channel, and the checkpoint all key on this path. 2 sublayers of 1 type
//! therefore never share 1 address. A checkpoint path is the layer path and the name of the array,
//! such as `2.forward.kernel`
//!
//! # Core components
//!
//! ## Layers
//! - **Dense**: fully connected layer with a configurable activation
//! - **Embedding**: trainable lookup table that turns whole-number indices into dense vectors
//! - **Reshaping**: `Flatten` collapses to rank 2, and `Reshape` rewrites the non-batch axes
//! - **Border**: `ZeroPadding1D/2D/3D` and `Cropping1D/2D/3D` resize the spatial axes at
//!   their ends
//! - **Upsampling**: `UpSampling1D/2D/3D` enlarge the spatial axes by a whole-number factor,
//!   with 5 interpolation modes on the 2D layer
//! - **Axis order**: `Permute` reorders the non-batch axes, `Reverse` flips the order inside
//!   1 axis, and `RepeatVector` repeats a feature vector into a sequence
//! - **Identity**: passes its input through unchanged, as a placeholder where a model needs a
//!   layer that does nothing
//! - **Rescaling**: applies `y = x * scale + offset` to each element before the trainable
//! layers
//! - **Activation**: standalone activation layers (ReLU, LeakyReLU, ELU, SELU, Sigmoid, Tanh, ...)
//! - **PReLU**: a ReLU whose negative-side slope is trainable, with a `shared_axes` rule that
//!   sets how many slopes it holds
//! - **Convolution**: 1D/2D/3D convolution, their transposed counterparts for a decoder, plus
//!   depthwise and separable variants in 1D and 2D
//! - **Pooling**: max and average pooling and their global variants for 1D, 2D, and 3D
//! - **Recurrent**: SimpleRNN, LSTM, and GRU sequence layers
//! - **Regularization**: dropout (including spatial), noise injection, and normalization layers,
//!   the last of which includes `UnitNormalization` for an L2 norm of 1 per group
//! - **Merge**: `Add`, `Subtract`, `Multiply`, `Average`, `Maximum`, and `Minimum` reduce several
//!   inputs element by element, and `Concatenate` joins them along 1 axis. Each layer takes
//!   several inputs and gives 1 output, so a graph model joins its branches with them.
//!   [`Concatenate::new`] never fails, and its axis counts against the full rank. The axis 0
//!   therefore joins the batch axis, and a negative axis counts back from the end. [`Average`] is
//!   a name shared by a merge layer and a classification averaging mode. The root of [`prelude`] keeps the
//!   averaging mode of the classification scores, so a caller that wants the layer must write
//!   `prelude::neural_network::Average` or `neural_network::layers::Average`
//!
//! ## Optimizers
//! - **SGD**: stochastic gradient descent with momentum
//! - **Adam**: adaptive moment estimation (coupled L2 weight decay)
//! - **AdamW**: Adam with decoupled weight decay
//! - **RMSprop**: root-mean-square propagation
//! - **AdaGrad**: adaptive gradient
//!
//! ## Loss functions
//! - **MeanSquaredError** / **MeanAbsoluteError**: regression
//! - **BinaryCrossEntropy**: binary classification
//! - **CategoricalCrossEntropy** / **SparseCategoricalCrossEntropy**: multi-class classification
//!
//! ## Shapes
//! - [`Shape`]: the shape of a tensor as a layer describes it, with a free axis for an extent that
//!   no configuration fixes. Every layer maps an input shape to an output shape through
//!   [`compute_output_shape`], which reads the layer configuration alone and answers before any
//!   tensor exists
//!
//! ## Weight initialization
//! - [`Initializer`]: how a layer draws the starting values of a weight array. Every drawn array of
//!   the module comes from this closed set of 3 rules. The layer supplies the [`Fans`] pair,
//!   because the kernel shape alone does not identify the convolution type
//!
//! ## Models
//! - [`Sequential`]: a linear stack of layers with an integrated training loop, prediction, and
//!   weight save and load. `fit` and `fit_with_batches` return a [`History`] with 1 loss value per
//!   epoch. Both build on the public `train_batch` step. `evaluate` scores a model without training
//!   it
//! - [`Graph`]: a directed graph of layers, with the same training loop, prediction, and weight
//!   save and load. It takes several inlets and gives several outlets, and it holds a layer arena.
//!   Several nodes can call 1 layer of that arena. Those nodes then share 1 set of arrays, and the
//!   sum of their gradients updates that set. [`GraphBuilder`] creates the model
//!
//! # Examples
//!
//! ## A chain of layers
//!
//! ```rust
//! use rustyml::neural_network::{
//!     Shape,
//!     sequential::SequentialBuilder,
//!     layers::{Activation, Dense},
//!     optimizers::Adam,
//!     losses::MeanSquaredError,
//! };
//! use ndarray::Array;
//!
//! // Create input and target tensors.
//! let x = Array::ones((2, 4)).into_dyn();  // 2 samples, 4 features
//! let y = Array::ones((2, 1)).into_dyn();  // 2 samples, 1 output
//!
//! // Build sequential model.
//! let mut model = SequentialBuilder::new()
//!     .add(Dense::new(8, Activation::ReLU).unwrap())   // Input layer: 4 -> 8
//!     .add(Dense::new(3, Activation::ReLU).unwrap())   // Hidden layer: 8 -> 3
//!     .add(Dense::new(1, Activation::Linear).unwrap()) // Output layer: 3 -> 1
//!     .build(&Shape::known(x.shape()))
//!     .unwrap();
//!
//! // Compile with optimizer and loss function.
//! model.compile(Adam::new(0.001, 0.9, 0.999, 1e-8, 0.0).unwrap(), MeanSquaredError::new());
//!
//! // Display model architecture.
//! model.summary();
//!
//! // Train the model.
//! model.fit(&x, &y, 100).unwrap();
//!
//! // Make predictions.
//! let predictions = model.predict(&x);
//! ```
//!
//! ## A graph of layers, joined by a merge layer
//!
//! ```rust
//! use rustyml::neural_network::{
//!     Shape,
//!     graph::GraphBuilder,
//!     layers::{Activation, Concatenate, Dense},
//!     optimizers::SGD,
//!     losses::MeanSquaredError,
//! };
//! use ndarray::Array;
//!
//! // 2 inlets of 4 samples each, and 1 target of 2 values per sample.
//! let left = Array::ones((4, 3)).into_dyn();
//! let right = Array::ones((4, 5)).into_dyn();
//! let y = Array::ones((4, 2)).into_dyn();
//!
//! // The 2 inlets keep registration order, and every call takes its tensors in that order.
//! let mut builder = GraphBuilder::new();
//! let a = builder.input(Shape::known(&[4, 3]));
//! let b = builder.input(Shape::known(&[4, 5]));
//! let joined = builder.add(Concatenate::new(-1), &[a, b]);
//! let head = builder.add(Dense::new(2, Activation::Linear).unwrap(), &[joined]);
//! let mut model = builder.build(&[head]).unwrap();
//!
//! // The merge layer adds the 2 feature counts, so the head reads 8 features.
//! assert_eq!(model.output_shapes()[0].to_string(), "(4, 2)");
//!
//! model.compile(SGD::new(0.01, 0.0, false, 0.0).unwrap(), MeanSquaredError::new());
//! model.fit(&[&left, &right], &[&y], 5).unwrap();
//!
//! let predictions = model.predict(&[&left, &right]).unwrap();
//! assert_eq!(predictions[0].shape(), &[4, 2]);
//! ```
//!
//! [`layers`]: crate::neural_network::layers
//! [`optimizers`]: crate::neural_network::optimizers
//! [`losses`]: crate::neural_network::losses
//! [`Sequential`]: crate::neural_network::sequential::Sequential
//! [`Graph`]: crate::neural_network::graph::Graph
//! [`traits`]: crate::neural_network::traits
//! [`Tensor`]: crate::neural_network::Tensor
//! [`forward`]: crate::neural_network::traits::UnaryLayer::forward
//! [`backward`]: crate::neural_network::traits::UnaryLayer::backward
//! [`Ctx`]: crate::neural_network::Ctx
//! [`Ctx::training`]: crate::neural_network::Ctx::training
//! [`Ctx::inference`]: crate::neural_network::Ctx::inference
//! [`LayerBase::sublayers`]: crate::neural_network::traits::LayerBase::sublayers
//! [`Ctx::sublayer`]: crate::neural_network::Ctx::sublayer
//! [`LayerPath`]: crate::neural_network::LayerPath
//! [`Concatenate::new`]: crate::neural_network::layers::Concatenate::new
//! [`Average`]: crate::neural_network::layers::Average
//! [`prelude`]: crate::prelude
//! [`Shape`]: crate::neural_network::Shape
//! [`compute_output_shape`]: crate::neural_network::traits::UnaryLayer::compute_output_shape
//! [`Initializer`]: crate::neural_network::Initializer
//! [`Fans`]: crate::neural_network::Fans
//! [`History`]: crate::neural_network::sequential::History
//! [`GraphBuilder`]: crate::neural_network::graph::GraphBuilder

use ndarray::ArrayD;

/// N-dimensional array used as a tensor in the neural network
pub type Tensor = ArrayD<f32>;

/// The per-pass context that carries caches, gradients and state between layers
pub mod ctx;
/// Neural-network error type, aggregated into the crate-wide `Error`
pub mod error;
/// A model whose layers form a directed graph, rather than a chain
pub mod graph;
/// How a layer draws the starting values of a weight array
pub mod initializer;
/// The address of 1 layer inside a model, and the walks over a tree of layers
pub mod layer_path;
/// Neural network layer implementations
pub mod layers;
/// Loss function implementations
pub mod losses;
/// Optimization algorithms for neural network training
pub mod optimizers;
/// Sequential model architecture
pub mod sequential;
/// The tensor shape a layer computes, with a free axis for an extent no configuration fixes
pub mod shape;
/// Trait interfaces for neural network models
pub mod traits;

pub use ctx::{Ctx, Grads, LayerId, StateSlot};
pub use error::NnError;
pub use initializer::{Fans, Initializer};
pub use layer_path::{LayerPath, Sublayer, SublayerMut, SublayerName};
pub use shape::Shape;