OptiRS NAS

Neural architecture search and hyperparameter optimization for the OptiRS machine-learning optimization library.

Scope

optirs-nas searches for optimizer architectures — which optimizer components to compose (SGD, Adam, AdamW, RMSprop, AdaGrad, Momentum, Lion, schedulers, regularizers) and with which hyperparameters — and provides the hyperparameter-search machinery around that. It is not a computer-vision layer-topology search: the search space is [nas_engine::SearchSpaceConfig], whose unit is an optimizer component.

Status: research-grade. Every algorithm listed below is implemented and tested in this crate, but the search-space and objective APIs may still change between releases. Validate a discovered configuration against your own baselines before relying on it.

Every Rust block in this file is compiled and run by cargo test --doc (see ReadmeDoctests in src/lib.rs), so the examples cannot drift away from the API.

Dependencies

[dependencies]
optirs-nas = "0.3.2"

scirs2-core is pulled in transitively; you do not need to depend on it yourself unless you use its types directly. The crate has no intra-workspace dependencies — it builds and tests standalone.

There are no Cargo features. Earlier versions declared bayesian, evolutionary, reinforcement, progressive, multi_objective and learned_integration; none of them gated any code (there was not a single cfg(feature = ...) in the crate), so they have been removed rather than left as documentation that promised a build knob that did not exist.

What is implemented

Multi-objective algorithms other than NSGA-II, NSGA-III, MOEA/D and weighted-sum (PAES, SPEA2, epsilon-constraint, goal programming) are not implemented. Configuring one returns OptimError::NotImplemented at engine construction instead of silently substituting NSGA-II or producing an empty Pareto front.

Running a search

use optirs_nas::nas_engine::resources::SystemResourceTracker;
use optirs_nas::nas_engine::telemetry::{FixedTelemetry, TelemetrySample};
use optirs_nas::nas_engine::{
    create_minimal_nas_config, NeuralArchitectureSearch, SearchStrategyType,
};
use std::time::Duration;

# fn main() -> Result<(), optirs_nas::error::OptimError> {
let mut config = create_minimal_nas_config::<f64>();
config.search_strategy = SearchStrategyType::Evolutionary;
config.search_budget = 2;
config.population_size = 4;

let mut engine = NeuralArchitectureSearch::new(config)?;

// The engine reports which strategy is actually running, not just what was requested.
assert_eq!(engine.search_strategy_name(), "EvolutionaryStrategy");

// Pin the telemetry so this example does not depend on the host's free memory.
// `resource_monitor_mut` is also the supported injection point for real telemetry.
engine.resource_monitor_mut().set_trackers(vec![Box::new(
    SystemResourceTracker::with_telemetry(
        "example".to_string(),
        Duration::from_secs(5),
        Box::new(FixedTelemetry::new("example", TelemetrySample::unknown())),
    ),
)]);

let results = engine.run_search()?;
assert!(!results.search_history.is_empty());
# Ok(())
# }

Every SearchStrategyType resolves to a real implementation. There is no "falls back to Evolutionary" arm.

Hyperparameter search

use optirs_nas::hyperparameter::{
    DistributionType, HyperparameterOptimizer, HyperparameterSpace, OptimizationStrategy,
    ParameterRange,
};

# fn main() -> Result<(), optirs_nas::error::OptimError> {
let mut space: HyperparameterSpace<f64> = HyperparameterSpace::new();
space.add_parameter(
    "learning_rate".to_string(),
    ParameterRange {
        name: "learning_rate".to_string(),
        min_value: 1e-5,
        max_value: 1e-1,
        distribution: DistributionType::Uniform,
        log_scale: true,
        discrete_values: None,
    },
);
space.add_categorical_parameter(
    "optimizer".to_string(),
    vec!["adam".to_string(), "sgd".to_string()],
);

// Grid search really enumerates the grid: 4 log-spaced rates x 2 optimizers.
let mut optimizer =
    HyperparameterOptimizer::with_seed(space, OptimizationStrategy::Grid, 7);
optimizer.set_grid_resolution(4);
assert_eq!(optimizer.grid_size(), Some(8));

let suggestion = optimizer.suggest_configuration()?;
let learning_rate = suggestion.parameters["learning_rate"];
assert!((1e-5..=1e-1).contains(&learning_rate));
# Ok(())
# }

Available strategies and what they do:

• Random — samples each parameter from its declared DistributionType (all five are implemented; none silently degrades to uniform).
• Grid — deterministic mixed-radix enumeration; the n-th call returns the n-th grid point and the whole grid is covered before it wraps. Continuous axes are log-spaced when the range declares log_scale.
• TPE — the published Tree-structured Parzen Estimator: the history is split by score quantile and two adaptive kernel-density estimators are fitted, candidates are drawn from the "good" density and ranked by the density ratio.
• Bayesian — a GP-free Nadaraya-Watson kernel-regression surrogate whose predictive variance grows away from the data, maximized under Expected Improvement / Probability of Improvement / Upper Confidence Bound / an entropy term.
• Evolutionary — tournament selection over a bounded elitist population, recombination per parameter, and Gaussian mutation on the parameter's own (log-aware) axis.

TPE, Bayesian and Evolutionary require a random initial design before they can be fitted — that is part of the published algorithms. During that phase last_strategy_used() reports Random, so the phase is visible rather than hidden:

use optirs_nas::hyperparameter::{
    DistributionType, HyperparameterOptimizer, HyperparameterSpace, OptimizationStrategy,
    ParameterRange,
};

# fn main() -> Result<(), optirs_nas::error::OptimError> {
let mut space: HyperparameterSpace<f64> = HyperparameterSpace::new();
space.add_parameter(
    "x".to_string(),
    ParameterRange {
        name: "x".to_string(),
        min_value: 0.0,
        max_value: 1.0,
        distribution: DistributionType::Uniform,
        log_scale: false,
        discrete_values: None,
    },
);

let mut optimizer =
    HyperparameterOptimizer::with_seed(space, OptimizationStrategy::TPE, 5);
let _first = optimizer.suggest_configuration()?;
assert_eq!(optimizer.last_strategy_used(), OptimizationStrategy::Random);
# Ok(())
# }

ParticleSwarm, SuccessiveHalving, Hyperband and BOHB are declared in the enum but not implemented; selecting one returns OptimError::NotImplemented.

Multi-objective search and the hypervolume indicator

The hypervolume is the exact indicator (recursive slicing / HSO), not a bounding-box heuristic, and it handles mixed Minimize / Maximize directions:

use optirs_nas::multi_objective::hypervolume_minimization;

// Union of [1,2]x[0,2] and [0,2]x[1,2] against the reference point (2, 2).
let front = vec![vec![1.0_f64, 0.0], vec![0.0, 1.0]];
let hypervolume = hypervolume_minimization(&front, &[2.0, 2.0]);
assert!((hypervolume - 3.0).abs() < 1e-12);

// A dominated point adds nothing; a genuinely non-dominated one strictly increases it.
let with_dominated = vec![vec![1.0_f64, 0.0], vec![0.0, 1.0], vec![1.5, 1.5]];
assert!((hypervolume_minimization(&with_dominated, &[2.0, 2.0]) - 3.0).abs() < 1e-12);
let with_new = vec![vec![1.0_f64, 0.0], vec![0.0, 1.0], vec![0.5, 0.5]];
assert!(hypervolume_minimization(&with_new, &[2.0, 2.0]) > 3.0);

HypervolumeCalculator wraps this with caching and a method selector. WFG, Quick and HSO are all exact and return identical values (documented on the enum); MonteCarlo is a real, seeded box-sampling estimator, not a hand-off to the exact routine.

Front quality metrics — convergence, objective_space_coverage, reference_distance, epsilon_dominance, spread, spacing — are all computed from the front ([multi_objective::metrics]). convergence is the relative change of the hypervolume since the previous update, measured against a reference point that is latched on first use so generations stay comparable.

Resource monitoring

Resource telemetry is injectable, and the default source reports only what it can honestly measure — std::thread::available_parallelism everywhere, plus /proc on Linux — and None for everything else (GPU count, disk capacity, network bandwidth, temperature, power). An unmeasurable resource is treated as unlimited, so a resource constraint can never abort a search on the strength of a value nobody measured:

use optirs_nas::nas_engine::telemetry::{StdTelemetry, TelemetrySource};

let sample = StdTelemetry::new().sample();

// Genuinely measurable.
assert!(sample.logical_cpus.is_some());

// Not reachable in pure Rust: reported as unknown, never invented.
assert!(sample.gpu_devices.is_none());
assert!(sample.temperature_celsius.is_none());
assert!(sample.power_watts.is_none());

To enforce GPU / thermal / power budgets, supply your own TelemetrySource and install it with NeuralArchitectureSearch::resource_monitor_mut().set_trackers(..) + SystemResourceTracker::with_telemetry(..).

Resource enforcement lives in exactly one place: the search loop samples the monitor each generation and returns OptimError::ResourceLimitExceeded on a violation. It never truncates a search silently.

Domain-specific search spaces

use optirs_nas::domain_specific_nas::{DomainNASEngine, DomainType};

# fn main() -> Result<(), optirs_nas::error::OptimError> {
let mut engine = DomainNASEngine::<f64>::new_for_domain(DomainType::ComputerVision);

// Search within the domain's pre-configured space, then check the winner against
// the domain's own structural rules.
let architectures = engine.search(4)?;
assert!(!architectures.is_empty());

if let Some(best) = engine.get_best_architecture() {
    let issues = engine.validate_for_domain(best)?;
    // `issues` is a (possibly empty) list of human-readable rule violations.
    assert!(issues.len() < 100);
}
# Ok(())
# }

Supported domains ([domain_specific_nas::DomainType]): ComputerVision, NaturalLanguageProcessing, TimeSeries, Reinforcement, Scientific, Multimodal. Constraints are expressed with [domain_specific_nas::NASConstraint] (MaxLatencyMs, MaxMemoryMb, MinAccuracy, RequiresComponent, MaxDepth, MaxWidth).

Notes on API shape

This crate is entirely synchronous. It contains no async fn, so nothing here is .awaited. Earlier revisions of this document showed await on every example; that was never valid.

Longer, runnable programs live in examples/ rather than in this file.

References

Techniques implemented here follow:

• Zoph & Le, Neural Architecture Search with Reinforcement Learning (2017)
• Liu et al., DARTS: Differentiable Architecture Search (2019)
• Deb et al., A Fast and Elitist Multiobjective Genetic Algorithm: NSGA-II (2002)
• Deb & Jain, An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I: NSGA-III (2014)
• Zhang & Li, MOEA/D: A Multiobjective Evolutionary Algorithm Based on Decomposition (2007)
• Bergstra, Bardenet, Bengio & Kegl, Algorithms for Hyper-Parameter Optimization (2011) — TPE
• While, Bradstreet & Barone, A Fast Way of Calculating Exact Hypervolumes (2012) — the exact slicing recursion

Contributing

OptiRS follows the Cool Japan organization's development standards. See the main OptiRS repository for contribution guidelines.

License

Apache-2.0