Crate optirs_core 

OptiRS Core - Advanced ML Optimization Built on SciRS2

Version: 0.3.2 Status: Pre-1.0 (0.3.x) - the public API may still change between 0.x releases

optirs-core provides state-of-the-art optimization algorithms for machine learning, built exclusively on the SciRS2 scientific computing ecosystem.

Dependencies

• scirs2-core 0.6.5, scirs2-optimize 0.6.5 - Required foundation
• scirs2-neural, scirs2-stats - Unconditional dependencies, pulled in for specific modules (e.g. neuromorphic, distribution-based regularizers) but not behind a feature
• scirs2-metrics - Optional, behind the metrics-integration feature
• scirs2-datasets - Optional, behind the cross-platform-testing feature

Quick Start

use optirs_core::optimizers::{Adam, Optimizer};
use scirs2_core::ndarray::Array1;

// Create optimizer
let mut optimizer = Adam::new(0.001);

// Prepare parameters and gradients
let params = Array1::from_vec(vec![1.0, 2.0, 3.0, 4.0]);
let grads = Array1::from_vec(vec![0.1, 0.2, 0.15, 0.08]);

// Perform optimization step
let updated_params = optimizer.step(&params, &grads)?;

Features

26 Optimizers

22 types implement the optimizers::Optimizer trait (optirs_core::optimizers, listed below). 4 more live in second_order (optirs_core::second_order) as a separate family not reachable through Optimizer: 2 implement second_order::SecondOrderOptimizer (Newton, and a second, independent L-BFGS implementation re-exported as SecondOrderLBFGS to avoid colliding with optimizers::LBFGS), and 2 expose their own inherent step/step_with_loss methods instead of a shared trait (NewtonCG, KFAC).

First-Order Methods (17):

• SGD - Stochastic Gradient Descent with optional momentum
• SimdSGD - SIMD-accelerated SGD
• Adam - Adaptive Moment Estimation
• AdamW - Adam with decoupled weight decay
• AdaDelta - Adaptive LR without manual tuning
• AdaBound - Smooth Adam→SGD transition
• Ranger - RAdam + Lookahead combination
• RMSprop - Root Mean Square Propagation
• Adagrad - Adaptive Gradient Algorithm
• LAMB - Layer-wise Adaptive Moments for Batch training
• LARS - Layer-wise Adaptive Rate Scaling
• Lion - Evolved Sign Momentum
• Lookahead - Look ahead optimizer wrapper
• RAdam - Rectified Adam
• SAM - Sharpness-Aware Minimization
• SparseAdam - Adam optimized for sparse gradients
• GroupedAdam - Adam with parameter groups

Quasi-Newton (1):

• LBFGS (optimizers::LBFGS) - Limited-memory BFGS with two-loop recursion

Meta-Learning Optimizers (4) - these also implement optimizers::Optimizer, so they drop into the same training loop as any other entry above:

• MAML - Model-Agnostic Meta-Learning (SecondOrder/FirstOrder/Reptile variants)
• MetaSGD - Meta-learned per-parameter learning rates
• ReptileOptimizer - First-order meta-learning
• NtmOptimizer - Neural Turing Machine-style memory-augmented optimizer

second_order module (4) - not part of optimizers::Optimizer:

• Newton (second_order::Newton) - implements SecondOrderOptimizer; diagonal Newton step with curvature flooring
• SecondOrderLBFGS (second_order::LBFGS) - implements SecondOrderOptimizer; a separate, simpler L-BFGS implementation from optimizers::LBFGS above
• NewtonCG - own step/step_with_loss API; Newton Conjugate Gradient with trust-region control
• KFAC - own step API; Kronecker-Factored Approximate Curvature

Performance Optimizations

SIMD Acceleration

use optirs_core::optimizers::{Optimizer, SimdSGD};
use scirs2_core::ndarray::Array1;

let params = Array1::from_elem(100_000, 1.0f32);
let grads = Array1::from_elem(100_000, 0.001f32);

let mut optimizer = SimdSGD::new(0.01f32);
let updated = optimizer.step(&params, &grads)?;

Parallel Processing

use optirs_core::optimizers::{Adam, Optimizer};
use optirs_core::parallel_optimizer::parallel_step_array1;
use scirs2_core::ndarray::Array1;

let params_list = vec![
    Array1::from_elem(10_000, 1.0),
    Array1::from_elem(20_000, 1.0),
];
let grads_list = vec![
    Array1::from_elem(10_000, 0.01),
    Array1::from_elem(20_000, 0.01),
];

let mut optimizer = Adam::new(0.001);
let results = parallel_step_array1(&mut optimizer, &params_list, &grads_list)?;

Memory-Efficient Operations

use optirs_core::memory_efficient_optimizer::GradientAccumulator;
use scirs2_core::ndarray::Array1;

let mut accumulator = GradientAccumulator::<f32>::new(1000);

// Accumulate gradients from micro-batches
for _ in 0..4 {
    let micro_grads = Array1::from_elem(1000, 0.1);
    accumulator.accumulate(&micro_grads.view())?;
}

let avg_grads = accumulator.average()?;

Production Metrics & Monitoring

use optirs_core::optimizer_metrics::{MetricsCollector, MetricsReporter};
use optirs_core::optimizers::{Adam, Optimizer};
use scirs2_core::ndarray::Array1;
use std::time::{Duration, Instant};

let mut collector = MetricsCollector::new();
collector.register_optimizer("adam");

let mut optimizer = Adam::new(0.001);
let params = Array1::from_elem(1000, 1.0);
let grads = Array1::from_elem(1000, 0.01);

let params_before = params.clone();
let start = Instant::now();
let params = optimizer.step(&params, &grads)?;
let duration = start.elapsed();

collector.update(
    "adam",
    duration,
    0.001,
    &grads.view(),
    &params_before.view(),
    &params.view(),
)?;

println!("{}", collector.summary_report());

Learning Rate Schedulers (optirs_core::schedulers)

• ConstantScheduler, ExponentialDecay, StepDecay, LinearDecay - basic decays
• CosineAnnealing, CosineAnnealingWarmRestarts - cosine schedules
• LinearWarmupDecay, OneCycle, CyclicLR - warmup / cyclic policies
• ReduceOnPlateau - metric-driven LR reduction
• CurriculumScheduler - staged curriculum transitions
• NoiseInjectionScheduler - stochastic LR perturbation
• AttentionAwareScheduler - component-specific LR scaling for Transformer models
• ViTLayerDecay - per-layer exponential LR decay for Vision Transformers
• CustomScheduler / CombinedScheduler / SchedulerBuilder - compose schedules from closures

Advanced Features

• Parameter Groups - Different learning rates per layer
• Gradient Accumulation - Micro-batch training for large models
• Gradient Clipping - Prevent exploding gradients
• Regularization - L1, L2, weight decay
• Privacy-Preserving - Differential privacy support (Rényi DP accountant, secure aggregation)
• Distributed Training - Parameter averaging, ring all-reduce/all-gather, pipeline parallelism (GPipe/1F1B), elastic (join/leave) training. Device-level GPU/TPU coordination lives in the separate optirs-gpu / optirs-tpu crates.

Architecture

SciRS2 Foundation

OptiRS-Core is built exclusively on the SciRS2 ecosystem:

• Arrays: Uses scirs2_core::ndarray (NOT direct ndarray)
• Random: Uses scirs2_core::random (NOT direct rand)
• SIMD: Uses scirs2_core::simd_ops for vectorization
• Parallel: Uses scirs2_core::parallel_ops for multi-core
• GPU: Built on scirs2_core::gpu abstractions
• Metrics: Uses scirs2_core::metrics for monitoring
• Error Handling: Uses scirs2_core::error::Result

This integration ensures:

• Type safety across the ecosystem
• Consistent performance optimizations
• Unified error handling
• Simplified dependency management

Module Organization

• optimizers - Core optimizer implementations
• schedulers - Learning rate scheduling
• simd_optimizer - SIMD-accelerated optimizers
• parallel_optimizer - Multi-core processing
• memory_efficient_optimizer - Memory optimization
• gpu_optimizer - GPU acceleration
• optimizer_metrics - Performance monitoring
• gradient_processing - Gradient manipulation
• regularizers - Regularization techniques
• second_order - Second-order methods
• distributed - Distributed training
• privacy - Privacy-preserving optimization

Performance

Benchmarks

All benchmarks use Criterion.rs with statistical analysis:

• optimizer_benchmarks - Compare optimizer implementations
• simd_benchmarks - SIMD vs scalar performance
• parallel_benchmarks - Multi-core scaling
• memory_efficient_benchmarks - Memory optimization impact
• gpu_benchmarks - GPU vs CPU comparison
• metrics_benchmarks - Monitoring overhead

Run benchmarks:

cargo bench --package optirs-core

Test Coverage

• 2202 tests - library + integration tests (cargo nextest run -p optirs-core --all-features)
• 95 doc tests - Documentation examples
• Zero clippy warnings - cargo clippy -p optirs-core --all-features --all-targets

Examples

See the examples/ directory for comprehensive examples:

• sgd_example.rs - Getting started
• advanced_optimization.rs - Schedulers, regularization, clipping
• performance_optimization.rs - SIMD, parallel, GPU acceleration
• production_monitoring.rs - Metrics and convergence detection

Contributing

When contributing, ensure:

• 100% SciRS2 usage - No direct ndarray/rand/rayon imports
• Zero clippy warnings - Run cargo clippy
• All tests pass - Run cargo test
• Documentation - Add examples to public APIs

License

licensed under Apache-2.0

Re-exports

pub use error::OptimError;

pub use error::OptimizerError;

pub use error::Result;

pub use unified_api::OptimizerConfig;

pub use unified_api::OptimizerFactory;

pub use unified_api::Parameter;

pub use unified_api::UnifiedOptimizer;

pub use adaptive_selection::AdaptiveOptimizerSelector;

pub use adaptive_selection::OptimizerStatistics;

pub use adaptive_selection::OptimizerType;

pub use adaptive_selection::PerformanceMetrics;

pub use adaptive_selection::ProblemCharacteristics;

pub use adaptive_selection::ProblemType;

pub use adaptive_selection::SelectionNetwork;

pub use adaptive_selection::SelectionStrategy;

pub use curriculum_optimization::AdaptiveCurriculum;

pub use curriculum_optimization::AdversarialAttack;

pub use curriculum_optimization::AdversarialConfig;

pub use curriculum_optimization::CurriculumManager;

pub use curriculum_optimization::CurriculumState;

pub use curriculum_optimization::CurriculumStrategy;

pub use curriculum_optimization::ImportanceWeightingStrategy;

pub use distributed::AveragingStrategy;

pub use distributed::CommunicationResult;

pub use distributed::CompressedGradient;

pub use distributed::CompressionStrategy;

pub use distributed::DistributedCoordinator;

pub use distributed::GradientCompressor;

pub use distributed::ParameterAverager;

pub use distributed::ParameterServer;

pub use domain_specific::CrossDomainKnowledge;

pub use domain_specific::DomainOptimizationConfig;

pub use domain_specific::DomainPerformanceMetrics;

pub use domain_specific::DomainRecommendation;

pub use domain_specific::DomainSpecificSelector;

pub use domain_specific::DomainStrategy;

pub use domain_specific::LearningRateScheduleType;

pub use domain_specific::OptimizationContext;

pub use domain_specific::RecommendationType;

pub use domain_specific::RegularizationApproach;

pub use domain_specific::ResourceConstraints;

pub use domain_specific::TrainingConfiguration;

pub use gpu_optimizer::GpuConfig;

pub use gpu_optimizer::GpuMemoryStats;

pub use gpu_optimizer::GpuOptimizer;

pub use gpu_optimizer::GpuUtils;

pub use gradient_accumulation::AccumulationMode;

pub use gradient_accumulation::GradientAccumulator as GradAccumulator;

pub use gradient_accumulation::MicroBatchTrainer;

pub use gradient_accumulation::VariableAccumulator;

pub use memory_efficient_optimizer::ChunkedOptimizer;

pub use memory_efficient_optimizer::GradientAccumulator as MemoryEfficientGradientAccumulator;

pub use memory_efficient_optimizer::MemoryUsageEstimator;

pub use neural_integration::architecture_aware::ArchitectureAwareOptimizer;

pub use neural_integration::architecture_aware::ArchitectureStrategy;

pub use neural_integration::forward_backward::BackwardHook;

pub use neural_integration::forward_backward::ForwardHook;

pub use neural_integration::forward_backward::NeuralIntegration;

pub use neural_integration::LayerArchitecture;

pub use neural_integration::LayerId;

pub use neural_integration::OptimizationConfig;

pub use neural_integration::ParamId;

pub use neural_integration::ParameterManager;

pub use neural_integration::ParameterMetadata;

pub use neural_integration::ParameterOptimizer;

pub use neural_integration::ParameterType;

pub use online_learning::ColumnGrowthStrategy;

pub use online_learning::LearningRateAdaptation;

pub use online_learning::LifelongOptimizer;

pub use online_learning::LifelongStats;

pub use online_learning::LifelongStrategy;

pub use online_learning::MemoryExample;

pub use online_learning::MemoryUpdateStrategy;

pub use online_learning::MirrorFunction;

pub use online_learning::OnlineLearningStrategy;

pub use online_learning::OnlineOptimizer;

pub use online_learning::OnlinePerformanceMetrics;

pub use online_learning::SharedKnowledge;

pub use online_learning::TaskGraph;

pub use optimizer_metrics::ConvergenceMetrics;

pub use optimizer_metrics::GradientStatistics;

pub use optimizer_metrics::MetricsCollector;

pub use optimizer_metrics::MetricsReporter;

pub use optimizer_metrics::OptimizerMetrics;

pub use optimizer_metrics::ParameterStatistics;

pub use parallel_optimizer::parallel_step;

pub use parallel_optimizer::parallel_step_array1;

pub use parallel_optimizer::ParallelBatchProcessor;

pub use parallel_optimizer::ParallelOptimizer;

pub use plugin::core::create_basic_capabilities;

pub use plugin::core::create_plugin_info;

pub use plugin::core::OptimizerPluginFactory;

pub use plugin::core::PluginCategory;

pub use plugin::core::PluginInfo;

pub use plugin::sdk::BaseOptimizerPlugin;

pub use plugin::OptimizerPlugin;

pub use plugin::PluginCapabilities;

pub use plugin::PluginLoader;

pub use plugin::PluginRegistry;

pub use plugin::PluginValidationFramework;

pub use privacy::AccountingMethod;

pub use privacy::ClippingStats;

pub use privacy::DifferentialPrivacyConfig;

pub use privacy::DifferentiallyPrivateOptimizer;

pub use privacy::MomentsAccountant;

pub use privacy::NoiseMechanism;

pub use privacy::PrivacyBudget;

pub use privacy::PrivacyValidation;

pub use quantum_inspired::HybridQuantumClassical;

pub use quantum_inspired::OptimizationPhase;

pub use quantum_inspired::QuantumAnnealing;

pub use quantum_inspired::QuantumOptimizerConfig;

pub use quantum_inspired::VariationalQuantumOptimizer;

pub use second_order::HessianInfo;

pub use second_order::Newton;

pub use second_order::NewtonCG;

pub use second_order::SecondOrderOptimizer;

pub use second_order::LBFGS as SecondOrderLBFGS;

pub use self_tuning::OptimizerInfo;

pub use self_tuning::OptimizerTrait;

pub use self_tuning::PerformanceStats;

pub use self_tuning::SelfTuningConfig;

pub use self_tuning::SelfTuningOptimizer;

pub use self_tuning::SelfTuningStatistics;

pub use self_tuning::TargetMetric;

pub use sensitivity_analysis::MorrisAnalyzer;

pub use sensitivity_analysis::MorrisIndices;

pub use sensitivity_analysis::OatAnalyzer;

pub use sensitivity_analysis::OatResult;

pub use sensitivity_analysis::SensitivityAnalyzer;

pub use sensitivity_analysis::SensitivityIndices;

pub use sensitivity_analysis::SobolAnalyzer;

pub use simd_optimizer::should_use_simd;

pub use simd_optimizer::SimdOptimizer;

pub use streaming::LearningRateAdaptation as StreamingLearningRateAdaptation;

pub use streaming::StreamingConfig;

pub use streaming::StreamingDataPoint;

pub use streaming::StreamingHealthStatus;

pub use streaming::StreamingMetrics;

pub use streaming::StreamingOptimizer;

pub use training_stabilization::AveragingMethod;

pub use training_stabilization::ModelEnsemble;

pub use training_stabilization::PolyakAverager;

pub use training_stabilization::WeightAverager;

pub use visualization::ColorScheme;

pub use visualization::ConvergenceInfo;

pub use visualization::DataSeries;

pub use visualization::MemoryStats as VisualizationMemoryStats;

pub use visualization::OptimizationMetric;

pub use visualization::OptimizationVisualizer;

pub use visualization::OptimizerComparison;

pub use visualization::PlotType;

pub use visualization::VisualizationConfig;

pub use optimizers::*;

pub use parameter_groups::*;

pub use regularizers::*;

pub use schedulers::*;

pub use gradient_processing::*;

pub use metrics::*;

Modules

adaptive_selection

benchmarking

coordination

curriculum_optimization

distributed

domain_specific

error

gpu_optimizer

GPU optimizer scaffolding

gradient_accumulation

gradient_flow

Gradient Flow Analysis Module

gradient_processing

hardware_aware

loss_landscape

Loss Landscape Analysis Module

memory_efficient

memory_efficient_optimizer

Memory-efficient optimizer operations

metrics

neural_integration

neuromorphic

online_learning

optimizer_composition

optimizer_metrics

Optimizer performance metrics and monitoring

optimizers

parallel_optimizer

Parallel optimizer operations using scirs2_core

parameter_groups

plugin

privacy

quantum_inspired

regularizers

reinforcement_learning

research

schedulers

second_order

self_tuning

sensitivity_analysis

simd_optimizer

SIMD-accelerated optimizer operations

streaming

training_stabilization

unified_api

utils

visualization

Macros

create_optimizer_plugin

Macro for creating a simple optimizer plugin

register_optimizer_plugin