optirs_learned/
lib.rs

1//! # OptiRS Learned - Learned Optimizers and Meta-Learning
2//!
3//! **Version:** 0.3.2
4//! **Status:** Research-grade implementations; APIs may still change between releases
5//!
6//! ⚠️ **Warning:** Learned optimizers are inherently sensitive to the distribution of
7//! tasks they were meta-trained on. Benchmark against `optirs-core`'s hand-designed
8//! optimizers on your own workload before depending on a learned one in production.
9//!
10//! `optirs-learned` provides learned optimizers, meta-learning algorithms, and adaptive
11//! optimization systems built on [SciRS2](https://github.com/cool-japan/scirs).
12//!
13//! ## Dependencies
14//!
15//! - `scirs2-core` 0.6.5 - Required foundation
16//! - `optirs-core` 0.3.2 - Core optimizers
17//!
18//! ## Implementation Status (v0.3.2)
19//!
20//! - ✅ Transformer-based optimizers ([`transformer`], [`transformer_based_optimizer`] -
21//!   self-/cross-attention over parameters)
22//! - ✅ LSTM optimizers ([`lstm`] - recurrent per-parameter update rule)
23//! - ✅ Meta-learning framework ([`meta_learning`] - MAML, Reptile, Meta-SGD)
24//! - ✅ Graph-neural-network optimizer ([`gnn_optimizer`]) and Neural Turing Machine
25//!   optimizer ([`ntm_optimizer`])
26//! - ✅ Differentiable optimizer search ([`darts_optimizer_search`] - DARTS-style)
27//! - ✅ Forward/reverse-mode autodiff engines ([`forward_mode`], [`reverse_mode`])
28//! - ✅ Continual learning ([`continual_learning`] - EWC, progressive networks), few-shot,
29//!   zero-shot and realtime drift adaptation ([`realtime_adaptation`])
30//! - 📝 Real, tested algorithms throughout - still labeled research-grade because learned
31//!   optimizers carry that distribution-sensitivity caveat, not because they are stubs
32//!
33//! ## Features
34//!
35//! ### Transformer-Based Optimizers
36//! - **Self-Attention** - Learn optimization patterns across parameters
37//! - **Cross-Attention** - Share optimization knowledge between layers
38//! - **Positional Encoding** - Parameter-aware optimization
39//! - **Multi-Head** - Diverse optimization strategies
40//!
41//! ### LSTM Optimizers
42//! - **Recurrent State** - Maintain long-term optimization memory
43//! - **Gating Mechanisms** - Adaptive learning rate control
44//! - **Sequence Modeling** - Learn optimization trajectories
45//! - **Stateful Updates** - Context-aware parameter updates
46//!
47//! ### Meta-Learning
48//! - **MAML** - Model-Agnostic Meta-Learning
49//! - **Reptile** - First-order meta-learning
50//! - **Meta-SGD** - Learn learning rates and update rules
51//! - **Task Adaptation** - Rapid fine-tuning on new tasks
52//!
53//! ### Few-Shot Optimization (`few_shot` module)
54//! - **PrototypicalNetwork** - Encode inputs into prototypes for rapid task identification
55//! - **FastAdaptationEngine** - Few-step adaptation with dynamic algorithm selection
56//! - **TaskSimilarityCalculator** / **EpisodicMemoryBank** - Find and reuse similar past tasks
57//! - **OnlineMAML** (`online_maml` module) - Continuous task-stream meta-learning with
58//!   staleness decay
59//!
60//! ## Example Usage
61//!
62//! Requires the `transformer` feature (on by default). Parameters and gradients are keyed
63//! by name, one entry per tensor, matching how a real training loop hands over named
64//! layers rather than a single flat array.
65//!
66//! ```rust,no_run
67//! use optirs_learned::transformer::{TransformerOptimizer, TransformerOptimizerConfig};
68//! use scirs2_core::ndarray::Array2;
69//! use std::collections::HashMap;
70//!
71//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
72//! // Create a transformer-based optimizer (default architecture)
73//! let config = TransformerOptimizerConfig::default();
74//! let mut optimizer = TransformerOptimizer::<f64>::new(config)?;
75//!
76//! let mut params = HashMap::new();
77//! params.insert("layer1.weight".to_string(), Array2::from_elem((4, 4), 1.0));
78//!
79//! let mut grads = HashMap::new();
80//! grads.insert("layer1.weight".to_string(), Array2::from_elem((4, 4), 0.01));
81//!
82//! // `step` takes the current loss and hands it back, so callers can chain steps
83//! let loss = optimizer.step(&mut params, &mut grads, 0.5)?;
84//! # Ok(())
85//! # }
86//! ```
87//!
88//! ## Feature Flags
89//!
90//! | Feature | Gates | Default |
91//! |---|---|---|
92//! | `transformer` | [`adaptive`], [`transformer`], [`transformer_based_optimizer`] modules and the [`TransformerOptimizer`] / `TransformerBasedOptimizer` re-exports | yes |
93//! | `lstm` | [`lstm`] module and the [`LSTMOptimizer`] re-export | yes |
94//! | `meta_learning` | [`meta_learning`] module (MAML, Reptile, Meta-SGD) | yes |
95//!
96//! All other modules ([`common`], [`continual_learning`], [`darts_optimizer_search`],
97//! [`forward_mode`], [`reverse_mode`], [`gnn_optimizer`], [`higher_order`],
98//! [`ntm_optimizer`], [`online_maml`], [`quantum_learned`], [`realtime_adaptation`],
99//! [`zero_shot`], `few_shot`, `es_meta_training`, `domain_objectives`, `domain_optimizers`,
100//! `episodic_memory_impl`, `cross_domain_transfer`) are ungated and always available.
101//! Four integration tests declare `required-features` in `Cargo.toml` (`lstm_layers`,
102//! `lstm_meta_training`: `lstm`; `adaptive_enhancement`: `transformer`;
103//! `xavier_initialization`: `lstm` + `transformer`), which is why
104//! `--no-default-features --features <one family>` is a real, smaller build.
105//!
106//! ## Architecture
107//!
108//! Built on [`optirs-core`](https://docs.rs/optirs-core) and `scirs2-core`
109//! (`scirs2_core::ndarray`, `scirs2_core::numeric`, `scirs2_core::random`) - no direct
110//! `ndarray`/`rand` dependency.
111//!
112//! ## References
113//!
114//! - Learning to Learn by Gradient Descent by Gradient Descent (Andrychowicz et al., 2016)
115//! - Learned Optimizers that Scale and Generalize (Metz et al., 2022)
116//! - VeLO: Training Versatile Learned Optimizers (Metz et al., 2023)
117//!
118//! ## Contributing
119//!
120//! Research contributions welcome! Follow SciRS2 integration guidelines.
121
122/// Adaptive architecture / performance-prediction layer for the transformer
123/// optimizers. Written directly against
124/// [`transformer_based_optimizer`], so it lives behind the same feature.
125#[cfg(feature = "transformer")]
126pub mod adaptive;
127pub mod common;
128pub mod continual_learning;
129pub mod cross_domain_transfer;
130pub mod darts_optimizer_search;
131pub mod domain_objectives;
132pub mod domain_optimizers;
133pub mod episodic_memory_impl;
134pub mod error;
135pub mod es_meta_training;
136pub mod few_shot;
137pub mod few_shot_impl;
138pub mod forward_mode;
139pub mod gnn_optimizer;
140pub mod higher_order;
141#[cfg(feature = "lstm")]
142pub mod lstm;
143#[cfg(feature = "meta_learning")]
144pub mod meta_learning;
145pub mod ntm_optimizer;
146pub mod online_maml;
147pub mod quantum_learned;
148pub mod realtime_adaptation;
149pub mod reverse_mode;
150#[cfg(feature = "transformer")]
151pub mod transformer;
152#[cfg(feature = "transformer")]
153pub mod transformer_based_optimizer;
154pub mod zero_shot;
155
156pub use common::{
157    LearnedOptimizerConfig, MetaOptimizationStrategy, NeuralOptimizerMetrics, NeuralOptimizerType,
158    OptimizerState, StateMetadata, TaskContext, TaskPerformance,
159};
160pub use continual_learning::{ElasticWeightConsolidation, NetworkColumn, ProgressiveNetworks};
161pub use darts_optimizer_search::{
162    ClosureObjective, DartsConfig, DartsOptimizerSearch, DifferentiableObjective,
163    DiscoveredOptimizer, PrimitiveHyperparams, PrimitiveState, QuadraticBowl, Rosenbrock,
164    SearchOutcome, UpdatePrimitive,
165};
166pub use domain_objectives::{MetaObjective, QuadraticObjective};
167pub use error::{OptimError, Result};
168pub use es_meta_training::{
169    EsMetaTrainer, MetaTrainable, MetaTrainingConfig, MetaTrainingReport, SelectionMetric,
170};
171pub use forward_mode::{DualNumber, ForwardModeEngine, ForwardModeStats, VectorDual};
172pub use gnn_optimizer::{
173    GnnOptimizer, GnnOptimizerConfig, GraphTopology, MessageActivation, MessageAggregation,
174};
175pub use higher_order::{
176    HessianConfig, HigherOrderConfig, HigherOrderEngine, HigherOrderStats, HvpMode, LayerInfo,
177    LayerType, MixedPartialMethod, MixedPartials, SparseHessian, ThirdOrderTensor,
178};
179#[cfg(feature = "lstm")]
180pub use lstm::LSTMOptimizer;
181pub use ntm_optimizer::{NtmOptimizer, NtmOptimizerConfig};
182pub use quantum_learned::{QuantumBackend, QuantumLearnedOptimizer};
183pub use realtime_adaptation::{
184    AdaptationDecision, AdaptationReason, DriftDetector, DriftDetectorConfig, DriftSignal,
185    RealtimeAdaptationConfig, RealtimeAdaptationController,
186};
187pub use reverse_mode::{GradientAccumulator, GradientContext, ReverseModeEngine, ReverseModeStats};
188#[cfg(feature = "transformer")]
189pub use transformer::TransformerOptimizer;
190#[cfg(feature = "transformer")]
191pub use transformer_based_optimizer::TransformerOptimizer as TransformerBasedOptimizer;
192pub use zero_shot::{
193    MetaExample, MetaFeatures, OptimizerHyperparameters, OptimizerKind, OptimizerRecommendation,
194    QuadraticProbe, RosenbrockProbe, TaskProbe, ZeroShotConfig, ZeroShotSelector,
195};
optirs_learned/lib.rs

optirs_learned/
lib.rs
