Expand description
§OptiRS Learned - Learned Optimizers and Meta-Learning
Version: 0.3.2 Status: Research-grade implementations; APIs may still change between releases
⚠️ Warning: Learned optimizers are inherently sensitive to the distribution of
tasks they were meta-trained on. Benchmark against optirs-core’s hand-designed
optimizers on your own workload before depending on a learned one in production.
optirs-learned provides learned optimizers, meta-learning algorithms, and adaptive
optimization systems built on SciRS2.
§Dependencies
scirs2-core0.6.5 - Required foundationoptirs-core0.3.2 - Core optimizers
§Implementation Status (v0.3.2)
- ✅ Transformer-based optimizers (
transformer,transformer_based_optimizer- self-/cross-attention over parameters) - ✅ LSTM optimizers (
lstm- recurrent per-parameter update rule) - ✅ Meta-learning framework (
meta_learning- MAML, Reptile, Meta-SGD) - ✅ Graph-neural-network optimizer (
gnn_optimizer) and Neural Turing Machine optimizer (ntm_optimizer) - ✅ Differentiable optimizer search (
darts_optimizer_search- DARTS-style) - ✅ Forward/reverse-mode autodiff engines (
forward_mode,reverse_mode) - ✅ Continual learning (
continual_learning- EWC, progressive networks), few-shot, zero-shot and realtime drift adaptation (realtime_adaptation) - 📝 Real, tested algorithms throughout - still labeled research-grade because learned optimizers carry that distribution-sensitivity caveat, not because they are stubs
§Features
§Transformer-Based Optimizers
- Self-Attention - Learn optimization patterns across parameters
- Cross-Attention - Share optimization knowledge between layers
- Positional Encoding - Parameter-aware optimization
- Multi-Head - Diverse optimization strategies
§LSTM Optimizers
- Recurrent State - Maintain long-term optimization memory
- Gating Mechanisms - Adaptive learning rate control
- Sequence Modeling - Learn optimization trajectories
- Stateful Updates - Context-aware parameter updates
§Meta-Learning
- MAML - Model-Agnostic Meta-Learning
- Reptile - First-order meta-learning
- Meta-SGD - Learn learning rates and update rules
- Task Adaptation - Rapid fine-tuning on new tasks
§Few-Shot Optimization (few_shot module)
- PrototypicalNetwork - Encode inputs into prototypes for rapid task identification
- FastAdaptationEngine - Few-step adaptation with dynamic algorithm selection
- TaskSimilarityCalculator / EpisodicMemoryBank - Find and reuse similar past tasks
- OnlineMAML (
online_mamlmodule) - Continuous task-stream meta-learning with staleness decay
§Example Usage
Requires the transformer feature (on by default). Parameters and gradients are keyed
by name, one entry per tensor, matching how a real training loop hands over named
layers rather than a single flat array.
use optirs_learned::transformer::{TransformerOptimizer, TransformerOptimizerConfig};
use scirs2_core::ndarray::Array2;
use std::collections::HashMap;
// Create a transformer-based optimizer (default architecture)
let config = TransformerOptimizerConfig::default();
let mut optimizer = TransformerOptimizer::<f64>::new(config)?;
let mut params = HashMap::new();
params.insert("layer1.weight".to_string(), Array2::from_elem((4, 4), 1.0));
let mut grads = HashMap::new();
grads.insert("layer1.weight".to_string(), Array2::from_elem((4, 4), 0.01));
// `step` takes the current loss and hands it back, so callers can chain steps
let loss = optimizer.step(&mut params, &mut grads, 0.5)?;§Feature Flags
| Feature | Gates | Default |
|---|---|---|
transformer | adaptive, transformer, transformer_based_optimizer modules and the TransformerOptimizer / TransformerBasedOptimizer re-exports | yes |
lstm | lstm module and the LSTMOptimizer re-export | yes |
meta_learning | meta_learning module (MAML, Reptile, Meta-SGD) | yes |
All other modules (common, continual_learning, darts_optimizer_search,
forward_mode, reverse_mode, gnn_optimizer, higher_order,
ntm_optimizer, online_maml, quantum_learned, realtime_adaptation,
zero_shot, few_shot, es_meta_training, domain_objectives, domain_optimizers,
episodic_memory_impl, cross_domain_transfer) are ungated and always available.
Four integration tests declare required-features in Cargo.toml (lstm_layers,
lstm_meta_training: lstm; adaptive_enhancement: transformer;
xavier_initialization: lstm + transformer), which is why
--no-default-features --features <one family> is a real, smaller build.
§Architecture
Built on optirs-core and scirs2-core
(scirs2_core::ndarray, scirs2_core::numeric, scirs2_core::random) - no direct
ndarray/rand dependency.
§References
- Learning to Learn by Gradient Descent by Gradient Descent (Andrychowicz et al., 2016)
- Learned Optimizers that Scale and Generalize (Metz et al., 2022)
- VeLO: Training Versatile Learned Optimizers (Metz et al., 2023)
§Contributing
Research contributions welcome! Follow SciRS2 integration guidelines.
Re-exports§
pub use common::LearnedOptimizerConfig;pub use common::MetaOptimizationStrategy;pub use common::NeuralOptimizerMetrics;pub use common::NeuralOptimizerType;pub use common::OptimizerState;pub use common::StateMetadata;pub use common::TaskContext;pub use common::TaskPerformance;pub use continual_learning::ElasticWeightConsolidation;pub use continual_learning::NetworkColumn;pub use continual_learning::ProgressiveNetworks;pub use darts_optimizer_search::ClosureObjective;pub use darts_optimizer_search::DartsConfig;pub use darts_optimizer_search::DartsOptimizerSearch;pub use darts_optimizer_search::DifferentiableObjective;pub use darts_optimizer_search::DiscoveredOptimizer;pub use darts_optimizer_search::PrimitiveHyperparams;pub use darts_optimizer_search::PrimitiveState;pub use darts_optimizer_search::QuadraticBowl;pub use darts_optimizer_search::Rosenbrock;pub use darts_optimizer_search::SearchOutcome;pub use darts_optimizer_search::UpdatePrimitive;pub use domain_objectives::MetaObjective;pub use domain_objectives::QuadraticObjective;pub use error::OptimError;pub use error::Result;pub use es_meta_training::EsMetaTrainer;pub use es_meta_training::MetaTrainable;pub use es_meta_training::MetaTrainingConfig;pub use es_meta_training::MetaTrainingReport;pub use es_meta_training::SelectionMetric;pub use forward_mode::DualNumber;pub use forward_mode::ForwardModeEngine;pub use forward_mode::ForwardModeStats;pub use forward_mode::VectorDual;pub use gnn_optimizer::GnnOptimizer;pub use gnn_optimizer::GnnOptimizerConfig;pub use gnn_optimizer::GraphTopology;pub use gnn_optimizer::MessageActivation;pub use gnn_optimizer::MessageAggregation;pub use higher_order::HessianConfig;pub use higher_order::HigherOrderConfig;pub use higher_order::HigherOrderEngine;pub use higher_order::HigherOrderStats;pub use higher_order::HvpMode;pub use higher_order::LayerInfo;pub use higher_order::LayerType;pub use higher_order::MixedPartialMethod;pub use higher_order::MixedPartials;pub use higher_order::SparseHessian;pub use higher_order::ThirdOrderTensor;pub use lstm::LSTMOptimizer;pub use ntm_optimizer::NtmOptimizer;pub use ntm_optimizer::NtmOptimizerConfig;pub use quantum_learned::QuantumBackend;pub use quantum_learned::QuantumLearnedOptimizer;pub use realtime_adaptation::AdaptationDecision;pub use realtime_adaptation::AdaptationReason;pub use realtime_adaptation::DriftDetector;pub use realtime_adaptation::DriftDetectorConfig;pub use realtime_adaptation::DriftSignal;pub use realtime_adaptation::RealtimeAdaptationConfig;pub use realtime_adaptation::RealtimeAdaptationController;pub use reverse_mode::GradientAccumulator;pub use reverse_mode::GradientContext;pub use reverse_mode::ReverseModeEngine;pub use reverse_mode::ReverseModeStats;pub use transformer::TransformerOptimizer;pub use transformer_based_optimizer::TransformerOptimizer as TransformerBasedOptimizer;pub use zero_shot::MetaExample;pub use zero_shot::MetaFeatures;pub use zero_shot::OptimizerHyperparameters;pub use zero_shot::OptimizerKind;pub use zero_shot::OptimizerRecommendation;pub use zero_shot::QuadraticProbe;pub use zero_shot::RosenbrockProbe;pub use zero_shot::TaskProbe;pub use zero_shot::ZeroShotConfig;pub use zero_shot::ZeroShotSelector;
Modules§
- adaptive
- Adaptive architecture / performance-prediction layer for the transformer
optimizers. Written directly against
transformer_based_optimizer, so it lives behind the same feature. Auto-generated module structure - common
- Common types and configurations for learned optimizers
- continual_
learning - cross_
domain_ transfer - Cross-domain knowledge transfer between domain optimizers.
- darts_
optimizer_ search - DARTS over Optimizer Update Rules
- domain_
objectives - Meta-training objectives: the tasks a learned optimizer is trained on.
- domain_
optimizers - Domain-Specific Optimizers
- episodic_
memory_ impl - error
- Error handling for OptiRS Learned
- es_
meta_ training - Meta-training learned optimizers by evolution strategies.
- few_
shot - few_
shot_ impl - forward_
mode - gnn_
optimizer - Message-Passing Graph Neural Network Optimizer
- higher_
order - lstm
- meta_
learning - Auto-generated module structure
- ntm_
optimizer - Neural Turing Machine (NTM) Memory-Augmented Learned Optimizer
- online_
maml - Online MAML for continuous task stream meta-learning.
- quantum_
learned - Quantum-inspired learned optimizers.
- realtime_
adaptation - Online drift-triggered hyperparameter adaptation.
- reverse_
mode - Reverse-mode automatic differentiation (backpropagation).
- transformer
- transformer_
based_ optimizer - zero_
shot - Zero-Shot Optimizer Selection