1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
//! Multimodal Neural Architecture Search.
//!
//! `multimodal_nas` provides a domain-specialised NAS engine for
//! *multimodal* models — architectures that ingest several input
//! modalities (image, text, audio), encode each one with a dedicated
//! per-modality sub-network, fuse the resulting representations with a
//! fusion operator, and emit a prediction through a shared post-fusion
//! head. It is the multimodal analogue of [`crate::speech_nas`].
//!
//! Unlike the generic NAS engine, the search space here is *structured*:
//! the architecture is split into one encoder sub-sequence per declared
//! modality, a single [`FusionOp`] that combines all encoders, and a head.
//! This structure makes the dominant validity rules explicit:
//!
//! * every declared modality must own a non-empty encoder *before* fusion
//! (an empty / missing encoder means fusion would run before that
//! modality is encoded);
//! * an encoder may only contain layers compatible with its modality
//! (generic [`MultimodalLayer::Dense`] / [`MultimodalLayer::Norm`] /
//! [`MultimodalLayer::Activation`] layers are allowed everywhere);
//! * the chosen fusion operator must be compatible with the number of
//! modalities present (for example [`FusionOp::Bilinear`] pools exactly
//! two modalities, while [`FusionOp::CrossAttention`] needs at least
//! two);
//! * no modality's encoder may *dwarf* the others beyond a configurable
//! capacity ratio (rough compute / parameter balance).
//!
//! All of these are checked by [`MultimodalArchitecture::validate`], which
//! returns a structured [`MultimodalValidationError`].
//!
//! # High-level workflow
//!
//! 1. Build a [`MultimodalSearchSpace`] (or take the default) describing
//! the available modalities, the per-modality candidate encoder layers,
//! the allowed fusion operators, the head layers, the length bounds, and
//! the capacity-balance ratio.
//! 2. Construct a [`MultimodalNasEngine`] from the search space.
//! 3. Repeatedly call [`MultimodalNasEngine::propose`] to draw fresh
//! architectures, [`MultimodalNasEngine::mutate`] to perturb a
//! known-good architecture, or [`MultimodalNasEngine::crossover`] to
//! recombine two parents. Every operation returns a `validate`-passing
//! architecture by construction (a final *rebalance* pass restores the
//! capacity balance after structural edits).
//! 4. Evaluate each proposed [`MultimodalArchitecture`] externally and feed
//! the result back via [`MultimodalNasEngine::record_evaluation`].
//! 5. Inspect [`MultimodalNasEngine::best`] /
//! [`MultimodalNasEngine::pareto_front`] to extract the search frontier.
//!
//! # Pareto frontier
//!
//! Multimodal model selection is multi-objective: higher accuracy may be
//! traded against latency or memory. [`MultimodalNasEngine::pareto_front`]
//! returns the set of *non-dominated* architectures over
//! `(accuracy, latency_ms, memory_mb)` — accuracy is *maximised* while
//! latency and memory are *minimised*.
//!
//! # Examples
//!
//! ```
//! use optirs_nas::multimodal_nas::{MultimodalEvaluation, MultimodalNasEngine};
//! use scirs2_core::random::Random;
//!
//! let mut engine = MultimodalNasEngine::with_default_search_space();
//! let mut rng = Random::seed(7);
//! let arch = engine.propose(&mut rng).expect("propose");
//! assert!(arch.validate().is_ok());
//! engine
//! .record_evaluation(
//! arch,
//! MultimodalEvaluation {
//! accuracy: 0.91,
//! latency_ms: 80.0,
//! memory_mb: 140.0,
//! param_count: 12_000_000,
//! },
//! )
//! .expect("record");
//! assert!(engine.best().is_some());
//! ```
pub use ;
pub use ;
pub use MultimodalSearchSpace;
pub use ;