Crate optirs_core 

OptiRS Core - Advanced ML Optimization Built on SciRS2

Version: 0.1.0 Status: Stable Release - Production Ready

optirs-core provides state-of-the-art optimization algorithms for machine learning, built exclusively on the SciRS2 scientific computing ecosystem.

Dependencies

• scirs2-core 0.1.1 - Required foundation

Quick Start

use optirs_core::optimizers::{Adam, Optimizer};
use scirs2_core::ndarray::Array1;

// Create optimizer
let mut optimizer = Adam::new(0.001);

// Prepare parameters and gradients
let params = Array1::from_vec(vec![1.0, 2.0, 3.0, 4.0]);
let grads = Array1::from_vec(vec![0.1, 0.2, 0.15, 0.08]);

// Perform optimization step
let updated_params = optimizer.step(&params, &grads)?;

Features

19 State-of-the-Art Optimizers

First-Order Methods:

• SGD - Stochastic Gradient Descent with optional momentum
• SimdSGD - SIMD-accelerated SGD (2-4x faster)
• Adam - Adaptive Moment Estimation
• AdamW - Adam with decoupled weight decay
• AdaDelta - Adaptive LR without manual tuning ⭐ NEW!
• AdaBound - Smooth Adam→SGD transition ⭐ NEW!
• Ranger - RAdam + Lookahead combination ⭐ NEW!
• RMSprop - Root Mean Square Propagation
• Adagrad - Adaptive Gradient Algorithm
• LAMB - Layer-wise Adaptive Moments for Batch training
• LARS - Layer-wise Adaptive Rate Scaling
• Lion - Evolved Sign Momentum
• Lookahead - Look ahead optimizer wrapper
• RAdam - Rectified Adam
• SAM - Sharpness-Aware Minimization
• SparseAdam - Adam optimized for sparse gradients
• GroupedAdam - Adam with parameter groups

Second-Order Methods:

• L-BFGS - Limited-memory BFGS
• K-FAC - Kronecker-Factored Approximate Curvature
• Newton-CG - Newton Conjugate Gradient ⭐ NEW!

Performance Optimizations (Phase 2 Complete)

SIMD Acceleration (2-4x speedup)

use optirs_core::optimizers::{Optimizer, SimdSGD};
use scirs2_core::ndarray::Array1;

let params = Array1::from_elem(100_000, 1.0f32);
let grads = Array1::from_elem(100_000, 0.001f32);

let mut optimizer = SimdSGD::new(0.01f32);
let updated = optimizer.step(&params, &grads)?;

Parallel Processing (4-8x speedup)

use optirs_core::optimizers::{Adam, Optimizer};
use optirs_core::parallel_optimizer::parallel_step_array1;
use scirs2_core::ndarray::Array1;

let params_list = vec![
    Array1::from_elem(10_000, 1.0),
    Array1::from_elem(20_000, 1.0),
];
let grads_list = vec![
    Array1::from_elem(10_000, 0.01),
    Array1::from_elem(20_000, 0.01),
];

let mut optimizer = Adam::new(0.001);
let results = parallel_step_array1(&mut optimizer, &params_list, &grads_list)?;

Memory-Efficient Operations

use optirs_core::memory_efficient_optimizer::GradientAccumulator;
use scirs2_core::ndarray::Array1;

let mut accumulator = GradientAccumulator::<f32>::new(1000);

// Accumulate gradients from micro-batches
for _ in 0..4 {
    let micro_grads = Array1::from_elem(1000, 0.1);
    accumulator.accumulate(&micro_grads.view())?;
}

let avg_grads = accumulator.average()?;

Production Metrics & Monitoring

use optirs_core::optimizer_metrics::{MetricsCollector, MetricsReporter};
use optirs_core::optimizers::{Adam, Optimizer};
use scirs2_core::ndarray::Array1;
use std::time::{Duration, Instant};

let mut collector = MetricsCollector::new();
collector.register_optimizer("adam");

let mut optimizer = Adam::new(0.001);
let params = Array1::from_elem(1000, 1.0);
let grads = Array1::from_elem(1000, 0.01);

let params_before = params.clone();
let start = Instant::now();
let params = optimizer.step(&params, &grads)?;
let duration = start.elapsed();

collector.update(
    "adam",
    duration,
    0.001,
    &grads.view(),
    &params_before.view(),
    &params.view(),
)?;

println!("{}", collector.summary_report());

Learning Rate Schedulers

• ExponentialDecay - Exponential learning rate decay
• StepDecay - Step-wise reduction
• CosineAnnealing - Cosine annealing schedule
• LinearWarmupDecay - Linear warmup with decay
• OneCycle - One cycle learning rate policy

Advanced Features

• Parameter Groups - Different learning rates per layer
• Gradient Accumulation - Micro-batch training for large models
• Gradient Clipping - Prevent exploding gradients
• Regularization - L1, L2, weight decay
• Privacy-Preserving - Differential privacy support
• Distributed Training - Multi-GPU and TPU coordination
• Neural Architecture Search - Automated architecture optimization

Architecture

SciRS2 Foundation

OptiRS-Core is built exclusively on the SciRS2 ecosystem:

• Arrays: Uses scirs2_core::ndarray (NOT direct ndarray)
• Random: Uses scirs2_core::random (NOT direct rand)
• SIMD: Uses scirs2_core::simd_ops for vectorization
• Parallel: Uses scirs2_core::parallel_ops for multi-core
• GPU: Built on scirs2_core::gpu abstractions
• Metrics: Uses scirs2_core::metrics for monitoring
• Error Handling: Uses scirs2_core::error::Result

This integration ensures:

• Type safety across the ecosystem
• Consistent performance optimizations
• Unified error handling
• Simplified dependency management

Module Organization

• optimizers - Core optimizer implementations
• schedulers - Learning rate scheduling
• simd_optimizer - SIMD-accelerated optimizers
• parallel_optimizer - Multi-core processing
• memory_efficient_optimizer - Memory optimization
• gpu_optimizer - GPU acceleration
• optimizer_metrics - Performance monitoring
• gradient_processing - Gradient manipulation
• regularizers - Regularization techniques
• second_order - Second-order methods
• distributed - Distributed training
• privacy - Privacy-preserving optimization

Performance

Benchmarks

All benchmarks use Criterion.rs with statistical analysis:

• optimizer_benchmarks - Compare all 16 optimizers
• simd_benchmarks - SIMD vs scalar performance
• parallel_benchmarks - Multi-core scaling
• memory_efficient_benchmarks - Memory optimization impact
• gpu_benchmarks - GPU vs CPU comparison
• metrics_benchmarks - Monitoring overhead

Run benchmarks:

cargo bench --package optirs-core

Test Coverage

• 549 unit tests - Core functionality
• 54 doc tests - Documentation examples
• 603 total tests - All passing
• Zero clippy warnings - Production quality

Examples

See the examples/ directory for comprehensive examples:

• basic_optimization.rs - Getting started
• advanced_optimization.rs - Schedulers, regularization, clipping
• performance_optimization.rs - SIMD, parallel, GPU acceleration
• production_monitoring.rs - Metrics and convergence detection

Contributing

When contributing, ensure:

• 100% SciRS2 usage - No direct ndarray/rand/rayon imports
• Zero clippy warnings - Run cargo clippy
• All tests pass - Run cargo test
• Documentation - Add examples to public APIs

License

Dual-licensed under MIT OR Apache-2.0

Re-exports

pub use error::OptimError;

pub use error::OptimizerError;

pub use error::Result;

pub use unified_api::OptimizerConfig;

pub use unified_api::OptimizerFactory;

pub use unified_api::Parameter;

pub use unified_api::UnifiedOptimizer;

pub use adaptive_selection::AdaptiveOptimizerSelector;

pub use adaptive_selection::OptimizerStatistics;

pub use adaptive_selection::OptimizerType;

pub use adaptive_selection::PerformanceMetrics;

pub use adaptive_selection::ProblemCharacteristics;

pub use adaptive_selection::ProblemType;

pub use adaptive_selection::SelectionNetwork;

pub use adaptive_selection::SelectionStrategy;

pub use curriculum_optimization::AdaptiveCurriculum;

pub use curriculum_optimization::AdversarialAttack;

pub use curriculum_optimization::AdversarialConfig;

pub use curriculum_optimization::CurriculumManager;

pub use curriculum_optimization::CurriculumState;

pub use curriculum_optimization::CurriculumStrategy;

pub use curriculum_optimization::ImportanceWeightingStrategy;

pub use distributed::AveragingStrategy;

pub use distributed::CommunicationResult;

pub use distributed::CompressedGradient;

pub use distributed::CompressionStrategy;

pub use distributed::DistributedCoordinator;

pub use distributed::GradientCompressor;

pub use distributed::ParameterAverager;

pub use distributed::ParameterServer;

pub use domain_specific::CrossDomainKnowledge;

pub use domain_specific::DomainOptimizationConfig;

pub use domain_specific::DomainPerformanceMetrics;

pub use domain_specific::DomainRecommendation;

pub use domain_specific::DomainSpecificSelector;

pub use domain_specific::DomainStrategy;

pub use domain_specific::LearningRateScheduleType;

pub use domain_specific::OptimizationContext;

pub use domain_specific::RecommendationType;

pub use domain_specific::RegularizationApproach;

pub use domain_specific::ResourceConstraints;

pub use domain_specific::TrainingConfiguration;

pub use gpu_optimizer::GpuConfig;

pub use gpu_optimizer::GpuMemoryStats;

pub use gpu_optimizer::GpuOptimizer;

pub use gpu_optimizer::GpuUtils;

pub use gradient_accumulation::AccumulationMode;

pub use gradient_accumulation::GradientAccumulator as GradAccumulator;

pub use gradient_accumulation::MicroBatchTrainer;

pub use gradient_accumulation::VariableAccumulator;

pub use memory_efficient_optimizer::ChunkedOptimizer;

pub use memory_efficient_optimizer::GradientAccumulator as MemoryEfficientGradientAccumulator;

pub use memory_efficient_optimizer::MemoryUsageEstimator;

pub use neural_integration::architecture_aware::ArchitectureAwareOptimizer;

pub use neural_integration::architecture_aware::ArchitectureStrategy;

pub use neural_integration::forward_backward::BackwardHook;

pub use neural_integration::forward_backward::ForwardHook;

pub use neural_integration::forward_backward::NeuralIntegration;

pub use neural_integration::LayerArchitecture;

pub use neural_integration::LayerId;

pub use neural_integration::OptimizationConfig;

pub use neural_integration::ParamId;

pub use neural_integration::ParameterManager;

pub use neural_integration::ParameterMetadata;

pub use neural_integration::ParameterOptimizer;

pub use neural_integration::ParameterType;

pub use online_learning::ColumnGrowthStrategy;

pub use online_learning::LearningRateAdaptation;

pub use online_learning::LifelongOptimizer;

pub use online_learning::LifelongStats;

pub use online_learning::LifelongStrategy;

pub use online_learning::MemoryExample;

pub use online_learning::MemoryUpdateStrategy;

pub use online_learning::MirrorFunction;

pub use online_learning::OnlineLearningStrategy;

pub use online_learning::OnlineOptimizer;

pub use online_learning::OnlinePerformanceMetrics;

pub use online_learning::SharedKnowledge;

pub use online_learning::TaskGraph;

pub use optimizer_metrics::ConvergenceMetrics;

pub use optimizer_metrics::GradientStatistics;

pub use optimizer_metrics::MetricsCollector;

pub use optimizer_metrics::MetricsReporter;

pub use optimizer_metrics::OptimizerMetrics;

pub use optimizer_metrics::ParameterStatistics;

pub use parallel_optimizer::parallel_step;

pub use parallel_optimizer::parallel_step_array1;

pub use parallel_optimizer::ParallelBatchProcessor;

pub use parallel_optimizer::ParallelOptimizer;

pub use plugin::core::create_basic_capabilities;

pub use plugin::core::create_plugin_info;

pub use plugin::core::OptimizerPluginFactory;

pub use plugin::core::PluginCategory;

pub use plugin::core::PluginInfo;

pub use plugin::sdk::BaseOptimizerPlugin;

pub use plugin::sdk::PluginTester;

pub use plugin::OptimizerPlugin;

pub use plugin::PluginCapabilities;

pub use plugin::PluginLoader;

pub use plugin::PluginRegistry;

pub use plugin::PluginValidationFramework;

pub use privacy::AccountingMethod;

pub use privacy::ClippingStats;

pub use privacy::DifferentialPrivacyConfig;

pub use privacy::DifferentiallyPrivateOptimizer;

pub use privacy::MomentsAccountant;

pub use privacy::NoiseMechanism;

pub use privacy::PrivacyBudget;

pub use privacy::PrivacyValidation;

pub use second_order::HessianInfo;

pub use second_order::Newton;

pub use second_order::NewtonCG;

pub use second_order::SecondOrderOptimizer;

pub use second_order::LBFGS as SecondOrderLBFGS;

pub use self_tuning::OptimizerInfo;

pub use self_tuning::OptimizerTrait;

pub use self_tuning::PerformanceStats;

pub use self_tuning::SelfTuningConfig;

pub use self_tuning::SelfTuningOptimizer;

pub use self_tuning::SelfTuningStatistics;

pub use self_tuning::TargetMetric;

pub use simd_optimizer::should_use_simd;

pub use simd_optimizer::SimdOptimizer;

pub use streaming::LearningRateAdaptation as StreamingLearningRateAdaptation;

pub use streaming::StreamingConfig;

pub use streaming::StreamingDataPoint;

pub use streaming::StreamingHealthStatus;

pub use streaming::StreamingMetrics;

pub use streaming::StreamingOptimizer;

pub use training_stabilization::AveragingMethod;

pub use training_stabilization::ModelEnsemble;

pub use training_stabilization::PolyakAverager;

pub use training_stabilization::WeightAverager;

pub use visualization::ColorScheme;

pub use visualization::ConvergenceInfo;

pub use visualization::DataSeries;

pub use visualization::MemoryStats as VisualizationMemoryStats;

pub use visualization::OptimizationMetric;

pub use visualization::OptimizationVisualizer;

pub use visualization::OptimizerComparison;

pub use visualization::PlotType;

pub use visualization::VisualizationConfig;

pub use optimizers::*;

pub use parameter_groups::*;

pub use regularizers::*;

pub use schedulers::*;

pub use gradient_processing::*;

Modules

adaptive_selection

benchmarking

coordination

curriculum_optimization

distributed

domain_specific

error

gpu_optimizer

GPU-accelerated optimizer operations

gradient_accumulation

gradient_processing

hardware_aware

memory_efficient

memory_efficient_optimizer

Memory-efficient optimizer operations

metrics

neural_integration

neuromorphic

online_learning

optimizer_composition

optimizer_metrics

Optimizer performance metrics and monitoring

optimizers

parallel_optimizer

Parallel optimizer operations using scirs2_core

parameter_groups

plugin

privacy

regularizers

research

schedulers

second_order

self_tuning

simd_optimizer

SIMD-accelerated optimizer operations

streaming

training_stabilization

unified_api

utils

visualization

Macros

create_optimizer_plugin

Macro for creating a simple optimizer plugin

register_optimizer_plugin