rustyml 0.16.0

//! Neural network primitives: layers, loss functions, optimizers, the 2 models, and the traits
//! that tie them together
//!
//! The module builds, trains, and runs neural networks.
//! Layers, optimizers, and losses live in the [`layers`](crate::neural_network::layers),
//! [`optimizers`](crate::neural_network::optimizers), and
//! [`losses`](crate::neural_network::losses) submodules.
//! [`Sequential`](crate::neural_network::sequential::Sequential) stacks layers into a
//! chain, and [`Graph`](crate::neural_network::graph::Graph) wires them into a directed graph.
//! The shared interfaces live in [`traits`](crate::neural_network::traits).
//! Every tensor in the framework is a [`Tensor`](crate::neural_network::Tensor), an `f32`
//! n-dimensional array
//!
//! A layer computes, and it holds no cache and no gradient.
//! [`forward`](crate::neural_network::traits::UnaryLayer::forward) and
//! [`backward`](crate::neural_network::traits::UnaryLayer::backward) take `&self` and a
//! [`Ctx`](crate::neural_network::Ctx). The context carries the training flag, the caches of the
//! pass, every parameter gradient, and the non-trainable state that a training pass changes.
//! [`Ctx::training`](crate::neural_network::Ctx::training) and
//! [`Ctx::inference`](crate::neural_network::Ctx::inference) pick the mode. A built model is
//! therefore `Send` and `Sync`, and several threads can run inference against 1 model
//!
//! # Core components
//!
//! ## Layers
//! - **Dense**: fully connected layer with a configurable activation
//! - **Embedding**: trainable lookup table that turns whole-number indices into dense vectors
//! - **Reshaping**: `Flatten` collapses to rank 2, and `Reshape` rewrites the non-batch axes
//! - **Border**: `ZeroPadding1D/2D/3D` and `Cropping1D/2D/3D` resize the spatial axes at
//!   their ends
//! - **Upsampling**: `UpSampling1D/2D/3D` enlarge the spatial axes by a whole-number factor,
//!   with 5 interpolation modes on the 2D layer
//! - **Axis order**: `Permute` reorders the non-batch axes, `Reverse` flips the order inside
//!   1 axis, and `RepeatVector` repeats a feature vector into a sequence
//! - **Identity**: passes its input through unchanged, as a placeholder where a model needs a
//!   layer that does nothing
//! - **Rescaling**: applies the fixed affine map `y = x * scale + offset` to every element, to
//!   condition an input in front of the first weight that a model trains
//! - **Activation**: standalone activation layers (ReLU, LeakyReLU, ELU, SELU, Sigmoid, Tanh, ...)
//! - **PReLU**: a ReLU whose negative-side slope is trainable, with a `shared_axes` rule that
//!   sets how many slopes it holds
//! - **Convolution**: 1D/2D/3D convolution, their transposed counterparts for a decoder, plus
//!   depthwise and separable variants in 1D and 2D
//! - **Pooling**: max and average pooling and their global variants for 1D, 2D, and 3D
//! - **Recurrent**: SimpleRNN, LSTM, and GRU sequence layers
//! - **Regularization**: dropout (including spatial), noise injection, and normalization layers,
//!   the last of which includes `UnitNormalization` for an L2 norm of 1 per group
//! - **Merge**: `Add`, `Subtract`, `Multiply`, `Average`, `Maximum`, and `Minimum` reduce several
//!   inputs element by element, and `Concatenate` joins them along 1 axis. Each layer takes
//!   several inputs and gives 1 output, so a graph model joins its branches with them.
//!   [`Concatenate::new`](crate::neural_network::layers::Concatenate::new) never fails, and its
//!   axis counts against the full rank. The axis 0 therefore joins the batch axis, and a
//!   negative axis counts back from the end.
//!   [`Average`](crate::neural_network::layers::Average) is the 1 name that 2 categories of the
//!   crate give to an item. The root of [`prelude`](crate::prelude) keeps the averaging mode of
//!   the classification scores, so a caller that wants the layer must write
//!   `prelude::neural_network::Average` or `neural_network::layers::Average`
//!
//! ## Optimizers
//! - **SGD**: stochastic gradient descent with momentum
//! - **Adam**: adaptive moment estimation (coupled L2 weight decay)
//! - **AdamW**: Adam with decoupled weight decay
//! - **RMSprop**: root-mean-square propagation
//! - **AdaGrad**: adaptive gradient
//!
//! ## Loss functions
//! - **MeanSquaredError** / **MeanAbsoluteError**: regression
//! - **BinaryCrossEntropy**: binary classification
//! - **CategoricalCrossEntropy** / **SparseCategoricalCrossEntropy**: multi-class classification
//!
//! ## Shapes
//! - [`Shape`](crate::neural_network::Shape): the shape of a tensor as a layer describes it,
//!   with a free axis for an extent that no configuration fixes. Every layer maps an input
//!   shape to an output shape through
//!   [`compute_output_shape`](crate::neural_network::traits::UnaryLayer::compute_output_shape),
//!   which reads the layer configuration alone and answers before any tensor exists
//!
//! ## Weight initialization
//! - [`Initializer`](crate::neural_network::Initializer): how a layer draws the starting values
//!   of a weight array. Every drawn array of the module comes from this closed set of 3 rules.
//!   The layer supplies the [`Fans`](crate::neural_network::Fans) pair, because a kernel shape
//!   alone cannot tell a plain convolution from a transposed one
//!
//! ## Models
//! - [`Sequential`](crate::neural_network::sequential::Sequential): a linear stack of layers with
//!   an integrated training loop, prediction, and weight save and load. `fit` and
//!   `fit_with_batches` return a [`History`](crate::neural_network::sequential::History) with 1
//!   loss value per epoch. Both build on the public `train_batch` step. `evaluate` scores a model
//!   without training it
//! - [`Graph`](crate::neural_network::graph::Graph): a directed graph of layers, with the same
//!   training loop, prediction, and weight save and load. It takes several inlets and gives
//!   several outlets, and it holds a layer arena. Several nodes can call 1 layer of that arena.
//!   Those nodes then share 1 set of arrays, and the sum of their gradients updates that set.
//!   [`GraphBuilder`](crate::neural_network::graph::GraphBuilder) is the only way to reach one
//!
//! # Examples
//!
//! ## A chain of layers
//!
//! ```rust
//! use rustyml::neural_network::{
//!     Shape,
//!     sequential::SequentialBuilder,
//!     layers::{Activation, Dense},
//!     optimizers::Adam,
//!     losses::MeanSquaredError,
//! };
//! use ndarray::Array;
//!
//! // Create input and target tensors
//! let x = Array::ones((2, 4)).into_dyn();  // 2 samples, 4 features
//! let y = Array::ones((2, 1)).into_dyn();  // 2 samples, 1 output
//!
//! // Build sequential model
//! let mut model = SequentialBuilder::new()
//!     .add(Dense::new(8, Activation::ReLU).unwrap())   // Input layer: 4 -> 8
//!     .add(Dense::new(3, Activation::ReLU).unwrap())   // Hidden layer: 8 -> 3
//!     .add(Dense::new(1, Activation::Linear).unwrap()) // Output layer: 3 -> 1
//!     .build(&Shape::known(x.shape()))
//!     .unwrap();
//!
//! // Compile with optimizer and loss function
//! model.compile(Adam::new(0.001, 0.9, 0.999, 1e-8, 0.0).unwrap(), MeanSquaredError::new());
//!
//! // Display model architecture
//! model.summary();
//!
//! // Train the model
//! model.fit(&x, &y, 100).unwrap();
//!
//! // Make predictions
//! let predictions = model.predict(&x);
//! ```
//!
//! ## A graph of layers, joined by a merge layer
//!
//! ```rust
//! use rustyml::neural_network::{
//!     Shape,
//!     graph::GraphBuilder,
//!     layers::{Activation, Concatenate, Dense},
//!     optimizers::SGD,
//!     losses::MeanSquaredError,
//! };
//! use ndarray::Array;
//!
//! // 2 inlets of 4 samples each, and 1 target of 2 values per sample
//! let left = Array::ones((4, 3)).into_dyn();
//! let right = Array::ones((4, 5)).into_dyn();
//! let y = Array::ones((4, 2)).into_dyn();
//!
//! // The 2 inlets keep registration order, and every call takes its tensors in that order
//! let mut builder = GraphBuilder::new();
//! let a = builder.input(Shape::known(&[4, 3]));
//! let b = builder.input(Shape::known(&[4, 5]));
//! let joined = builder.add(Concatenate::new(-1), &[a, b]);
//! let head = builder.add(Dense::new(2, Activation::Linear).unwrap(), &[joined]);
//! let mut model = builder.build(&[head]).unwrap();
//!
//! // The merge layer adds the 2 feature counts, so the head reads 8 features
//! assert_eq!(model.output_shapes()[0].to_string(), "(4, 2)");
//!
//! model.compile(SGD::new(0.01, 0.0, false, 0.0).unwrap(), MeanSquaredError::new());
//! model.fit(&[&left, &right], &[&y], 5).unwrap();
//!
//! let predictions = model.predict(&[&left, &right]).unwrap();
//! assert_eq!(predictions[0].shape(), &[4, 2]);
//! ```

use ndarray::ArrayD;

/// N-dimensional array used as a tensor in the neural network
pub type Tensor = ArrayD<f32>;

/// The per-pass context that carries caches, gradients and state between layers
pub mod ctx;
/// Neural-network error type, aggregated into the crate-wide [`Error`](crate::error::Error)
pub mod error;
/// A model whose layers form a directed graph, rather than a chain
pub mod graph;
/// How a layer draws the starting values of a weight array
pub mod initializer;
/// Neural network layer implementations
pub mod layers;
/// Loss function implementations
pub mod losses;
/// Optimization algorithms for neural network training
pub mod optimizers;
/// Sequential model architecture
pub mod sequential;
/// The tensor shape a layer computes, with a free axis for an extent no configuration fixes
pub mod shape;
/// Trait interfaces for neural network models
pub mod traits;

pub use ctx::{Ctx, Grads, LayerId, StateSlot};
pub use error::NnError;
pub use initializer::{Fans, Initializer};
pub use shape::Shape;