scivex-nn

Neural networks for Scivex. Autograd engine, layers, optimizers, and loss functions for building and training deep learning models.

Highlights

• Autograd — Automatic differentiation with dynamic computation graph
• Layers — Linear, Conv1d/2d/3d, RNN, LSTM, GRU, BatchNorm, LayerNorm, Dropout
• Attention — MultiHeadAttention, MultiQuery, GroupedQuery, Flash, Rotary/Sinusoidal positional encoding
• Transformer — TransformerEncoderLayer, TransformerDecoderLayer
• Pooling — MaxPool1d/2d, AvgPool1d/2d, AdaptiveAvgPool, GlobalAvgPool
• Activations — ReLU, GELU, SiLU, Sigmoid, Tanh, Softmax, LeakyReLU, ELU
• Optimizers — SGD (with momentum), Adam, AdamW, RMSprop, Adagrad
• Losses — MSE, CrossEntropy, BinaryCrossEntropy, Huber, L1
• Schedulers — StepLR, CosineAnnealing, ReduceOnPlateau, WarmupCosine
• Mixed precision — FP16/BF16 support via scivex-core half types
• GPU support — Optional GPU acceleration via scivex-gpu

Usage

use scivex_nn::prelude::*;

let mut model = Sequential::new(vec![
    Box::new(Linear::new(784, 256, &mut rng)),
    Box::new(ReLU),
    Box::new(Linear::new(256, 10, &mut rng)),
]);

let mut optimizer = Adam::new(model.parameters(), 0.001);
let loss = cross_entropy_loss(&logits, &targets);
loss.backward(None);
optimizer.step();

License

MIT