Crate torsh_optim 

Optimization algorithms for ToRSh

This crate provides PyTorch-compatible optimizers built on top of scirs2-optim.

Features

• 80+ optimizers: Comprehensive collection including Adam, SGD, RAdam, Ranger, Lion, Sophia, and more
• Modern optimizers: Latest research including Schedule-Free AdamW and Prodigy
• Second-order methods: L-BFGS, Newton-CG, Trust Region, K-FAC, AdaHessian
• Learning rate schedulers: Step, exponential, cosine annealing, one-cycle, and more
• Mixed precision training: Full fp16/fp32 support with loss scaling
• Distributed optimization: AsyncSGD, Elastic Averaging, Federated Learning
• Advanced features: Gradient accumulation, fused kernels, memory-efficient implementations
• Research features: Quantum-inspired, neuromorphic, continual learning, green AI optimizers

Quick Start

use torsh_optim::prelude::*;
use torsh_tensor::Tensor;
use std::sync::Arc;
use parking_lot::RwLock;

// Create parameters
let params = vec![Arc::new(RwLock::new(Tensor::scalar(1.0)?))];

// Create optimizer
let mut optimizer = Adam::new(params, Some(0.001), None, None, None, false);

// Training loop
for _ in 0..100 {
    // ... compute gradients ...
    optimizer.step()?;
    optimizer.zero_grad();
}

Re-exports

pub use adam::Adam;

pub use adam::AdamW;

pub use distributed::DistributedBackend;

pub use distributed::DistributedConfig;

pub use distributed::DistributedOptimizer;

pub use distributed::SyncStrategy;

pub use rmsprop::RMSprop;

pub use sgd::SGD;

Modules

adabelief

AdaBelief optimizer implementation

adabound

AdaBound optimizer implementation

adadelta

AdaDelta optimizer

adagrad

AdaGrad optimizer

adahessian

AdaHessian optimizer

adam

Adam and AdamW optimizers

adamax

AdaMax optimizer implementation

advanced

Advanced optimizers using SciRS2 optimization algorithms

asgd

Averaged Stochastic Gradient Descent optimizer

bayesian_optimization

Bayesian Optimization for Hyperparameter Tuning

benchmarks

Comprehensive benchmarking suite for optimizers

checkpointing

composition

Optimizer composition tools

continual_learning

Continual Learning Optimizers

cross_framework_validation

Cross-framework validation tests for ToRSh optimizers

debugging

Debugging and analysis tools for optimizers

differential_privacy

Differential Privacy support for optimizers

distributed

Distributed optimization for multi-process training

evolutionary_strategies

Evolutionary Strategies for Optimization

ftrl

FTRL (Follow-The-Regularized-Leader) optimizer

fused_kernels

Fused optimizer kernels for improved performance

grad_accumulation

Gradient accumulation utilities for optimizers

gradient_free

Gradient-Free Optimization Methods

green_ai

Green AI Optimizers

hyperparameter_tuning

Automatic hyperparameter tuning for optimizers

kfac

K-FAC (Kronecker-Factored Approximate Curvature) optimizer

lamb

LAMB (Large Batch Optimization for Deep Learning) optimizer implementation

lazy_updates

lbfgs

Limited-memory BFGS optimizer

lion

Lion (Evolved Sign Momentum) optimizer

lookahead

Lookahead optimizer implementation

low_precision

lr_scheduler

Learning rate schedulers

lr_scheduler_additional

Additional learning rate schedulers

lr_scheduler_enhanced

Enhanced learning rate schedulers with advanced features

memory_efficient

Memory-efficient optimizer implementations

memory_mapped

mixed_precision

Mixed precision support for optimizers

nadam

NAdam optimizer implementation

natural_gradient

Natural Gradient optimizer

neural_optimizer

Neural Optimizer - Research Feature

neuromorphic

Neuromorphic Optimization

newton_cg

Newton-CG (Newton-Conjugate Gradient) optimizer

numerical_stability_tests

Numerical stability tests for optimizers

online_learning

Online learning optimizers and variance-reduced methods

optimizer

Base optimizer implementation utilities

prelude

prodigy

Prodigy (An Adaptive Learning Rate Method) optimizer

quantum_inspired

Quantum-Inspired Optimization Algorithms

radam

Rectified Adam optimizer

ranger

Ranger optimizer implementation

rmsprop

RMSprop (Root Mean Square Propagation) optimizer

robustness

Robustness features for optimizers

rprop

Resilient Backpropagation optimizer

schedule_free

Schedule-Free Optimizers

sgd

Stochastic Gradient Descent (SGD) optimizer

shampoo

Shampoo optimizer

sophia

Sophia (Second-order Clipped Stochastic Optimization) optimizer

sparse_adam

Sparse Adam optimizer

sparse_updates

state_dict_ops

Optimized state dict operations for efficient optimizer state management

stress_tests

Stress tests for ToRSh optimizers

trust_region

Trust Region optimization methods

yellowfin

YellowFin optimizer

Macros

impl_base_scheduler_methods

Macro to implement common LRScheduler methods for schedulers with a base field

impl_scheduler_with_state

Macro to implement common LRScheduler methods with custom state handling

Structs

OptimizerOptions

Common optimizer options

OptimizerState

Optimizer state for serialization

ParamGroup

Parameter group

ParamGroupBuilder

Builder for creating parameter groups with various options

ParamGroupState

Parameter group state

Enums

OptimizerError

Optimizer-specific error type

Constants

VERSION

VERSION_MAJOR

VERSION_MINOR

VERSION_PATCH

Traits

Optimizer

Base optimizer trait

Type Aliases

OptimizerResult

Result type for optimizer operations