mathverse-ai
AI/ML mathematical primitives: tensors, activations, losses, optimizers, and attention mechanisms — all in pure Rust with zero external dependencies.
Features
- N-dimensional tensor with broadcasting, reshaping, and element-wise math
- Activation functions: ReLU, GELU, Swish, Mish, Softmax, Sigmoid, Tanh, and more
- Loss functions: MSE, MAE, Huber, Cross-Entropy, Binary Cross-Entropy, KL Divergence, Hinge
- Evaluation metrics: Accuracy, Precision, Recall, F1, Confusion Matrix, ROC AUC, R², Explained Variance
- Stateful optimizers: SGD (with momentum), Adam, AdamW
- Learning rate schedulers: Constant, Step Decay, Cosine Annealing, Linear Warmup
- Attention math: QKV projection, scaled dot-product attention, multi-head attention
- Positional encodings: sinusoidal (additive), rotary (RoPE)
- Layer/Batch/RMS normalization
Module Overview
| Module | Purpose | Key Functions |
|---|---|---|
tensor |
N-D tensor with row-major layout, broadcasting, math ops | Tensor::zeros, matmul, add, layer_norm, broadcast_to |
activations |
Element-wise activation functions and derivatives | relu, gelu, sigmoid, softmax, swish, mish |
losses |
Regression and classification loss functions | mse, cross_entropy, binary_cross_entropy_with_logits, huber |
metrics |
Model evaluation metrics | accuracy, f1, confusion_matrix, roc_auc, r_squared |
optimizers |
Stateful gradient-based optimizers | Sgd, Adam, AdamW, LrScheduler |
attention |
Transformer attention math | scaled_dot_product_attention, multi_head_attention, apply_rope |
Installation
Or add to Cargo.toml:
[]
= { = "../mathverse-ai" }
Quick Start
use ;
Module Documentation
Tensor
The core N-dimensional array type. Supports broadcasting (NumPy-style trailing dimensions), element-wise arithmetic, matrix multiply, reductions, normalization, and shape manipulation.
Memory Layout (row-major):
┌──────────────────────────────────────┐
│ Shape: [2, 3, 4] │
│ Data: [a₀₀₀ a₀₀₁ a₀₀₂ a₀₀₃ │
│ a₀₁₀ a₀₁₁ a₀₁₂ a₀₁₃ │
│ a₀₂₀ a₀₂₁ a₀₂₂ a₀₂₃ │
│ a₁₀₀ a₁₀₁ a₁₀₂ a₁₀₃ │
│ a₁₁₀ a₁₁₁ a₁₁₂ a₁₁₃ │
│ a₁₂₀ a₁₂₁ a₁₂₂ a₁₂₃] │
│ Numel: 24 │
└──────────────────────────────────────┘
use Tensor;
// Constructors
let t = zeros;
let t = arange.reshape.unwrap;
// Broadcasting
let a = new.unwrap;
let b = new.unwrap;
let c = a.add.unwrap; // shape [2, 3]
// c = [[11, 21, 31],
// [12, 22, 32]]
// Matrix multiply
let a = new.unwrap;
let b = new.unwrap;
let c = a.matmul.unwrap; // shape [2, 2]
// c = [[58, 64],
// [139, 154]]
// Layer normalization
let t = new.unwrap;
let norm = t.layer_norm; // each row has mean ≈ 0
Real-world use: Data preprocessing, neural network forward passes, batch operations.
Activations
All activation functions operate element-wise on tensors. Derivative functions (suffixed _grad) are provided for backpropagation.
Activation Comparison:
ReLU GELU Swish Sigmoid
┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐
│ /│ │ / │ │ / │ │ │
│ / │ │ / │ │ _/ │ │───────/│
────┼──────/─┼────┼────/───┼────┼──/─────┼────┼──────/─┼───
│ / │ │ / │ │ / │ │ / │
│ / │ │ / │ │/ │ │ / │
└────────┘ └────────┘ └────────┘ └────────┘
f(x)=max(0,x) f(x)=0.5x(1+tanh) f(x)=x·σ(x) f(x)=1/(1+e⁻ˣ)
use ;
let t = new.unwrap;
let r = relu;
// [0.0, 0.0, 0.0, 1.0, 2.0]
let s = sigmoid;
// [0.1192, 0.2689, 0.5, 0.7311, 0.8808]
// Softmax (probabilities sum to 1)
let logits = new.unwrap;
let probs = softmax.unwrap;
// [0.0900, 0.2447, 0.6652]
let g = gelu;
// GELU is smoother than ReLU, used in GPT/BERT
Real-world use: Hidden layer activations in neural networks. GELU in transformers, ReLU in CNNs, Sigmoid for binary output.
Loss Functions
Compute scalar losses and their gradients for training neural networks.
use ;
// Regression: MSE
let pred = new.unwrap;
let target = new.unwrap;
println!;
// MAE: 1.3333
// Classification: Cross-Entropy
let logits = new.unwrap;
let targets = new.unwrap;
println!;
// CE: 0.0001
// Binary: BCE with logits (numerically stable)
let logits = new.unwrap;
let target = new.unwrap;
println!;
// BCE: 0.0023
Formulas:
- MSE:
L = mean((pred - target)²) - Cross-Entropy:
L = -mean(log_softmax(logits)[target]) - Huber:
L = 0.5(p-t)² if |p-t|≤δ, else δ(|p-t| - 0.5δ)
Metrics
use ;
let pred = new.unwrap;
let target = new.unwrap;
println!;
// Accuracy: 0.60
let f1_scores = f1.unwrap;
// Per-class F1 scores
let cm = confusion_matrix.unwrap;
// 3×3 confusion matrix
Optimizers
Stateful optimizers that maintain internal momentum/adaptation state.
Optimizer Convergence (minimizing f(x) = x²):
x
10│*
│ *
│ *
│ * Adam (fast)
│ ·····
5 │ · SGD
│ · (slower)
│ ········
│ ·············
0 └───────────────────────
0 50 100 150 steps
use ;
// Adam optimizer
let mut opt = new;
let mut params = vec!;
for _ in 0..1000
// params ≈ [0, 0, 0]
// Cosine annealing LR schedule
let mut scheduler = new;
for _ in 0..100
// LR decays from 1.0 → 0.0001 following cosine curve
Attention
Transformer attention mechanisms: QKV projection, scaled dot-product attention, multi-head attention, and positional encodings.
Multi-Head Attention:
┌─────┐
X ──────────────┤Wq,Wk│──── Q,K,V
[batch,seq,d] │Wv │
└──┬──┘
│
┌────────────┼────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Head 1 │ │ Head 2 │ │ Head h │
│ Attn(Q₁, │ │ Attn(Q₂, │ │ Attn(Qₕ, │
│ K₁,V₁) │ │ K₂,V₂) │ │ Kₕ,Vₕ) │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
└────────┬───┘────────────┘
│ Concat
┌───▼───┐
│ Wo │
└───┬───┘
▼
Output
use ;
// Scaled dot-product attention
let q = new.unwrap;
let k = new.unwrap;
let v = new.unwrap;
let = scaled_dot_product_attention.unwrap;
// out: [1, 2, 4], weights: [1, 2, 2] (softmax attention map)
// Sinusoidal positional encoding
let enc = sinusoidal_encoding;
// enc shape: [1, 10, 64]
// RoPE (Rotary Position Embeddings)
let x = randn;
let y = apply_rope;
// Norm-preserving rotation by position
Formulas:
- Attention:
Attention(Q,K,V) = softmax(QKᵀ / √dₖ) V - RoPE:
θ_i = pos / base^(2i/d), apply 2D rotation to each pair
Future Scope
- Autograd / automatic differentiation
- GPU acceleration via
wgpucompute shaders - Convolution operations (Conv1d, Conv2d)
- Dropout, LayerNorm with learnable parameters
- KV-cache for autoregressive inference
- Flash attention for memory-efficient attention
- Mixed precision (f16/bf16) support
License
MIT OR Apache-2.0