Struct TransformerOptimizerConfig 

pub struct TransformerOptimizerConfig {

    pub base_config: LearnedOptimizerConfig,
    pub modeldim: usize,
    pub numheads: usize,
    pub ff_dim: usize,
    pub num_layers: usize,
    pub max_sequence_length: usize,
    pub attention_dropout: f64,
    pub ff_dropout: f64,
    pub layer_norm_eps: f64,
    pub pre_layer_norm: bool,
    pub pos_encoding_type: PositionalEncodingType,
    pub relative_position_bias: bool,
    pub use_rope: bool,
    pub causal_attention: bool,
    pub activation: ActivationFunction,
    pub dropout_seed: u64,
    pub update_modulation: f64,
    pub gradient_checkpointing: bool,
    pub attention_optimization: AttentionOptimization,
    pub multi_scale_attention: bool,
    pub cross_attention: bool,
    pub memory_efficient: bool,
}

Configuration specific to Transformer optimizer

Fields

base_config: LearnedOptimizerConfig

Base learned optimizer config

modeldim: usize

Model dimension (d_model)

numheads: usize

Number of attention heads

ff_dim: usize

Feed-forward network dimension

num_layers: usize

Number of transformer layers

max_sequence_length: usize

Maximum sequence length, which is also the optimization-history window fed to the network on every step

attention_dropout: f64

Attention dropout rate

ff_dropout: f64

Feed-forward dropout rate

layer_norm_eps: f64

Layer normalization epsilon

pre_layer_norm: bool

Use pre-layer normalization

pos_encoding_type: PositionalEncodingType

Positional encoding type

relative_position_bias: bool

Enable relative position bias

use_rope: bool

Use rotary position embedding

causal_attention: bool

Mask out future positions in self-attention

activation: ActivationFunction

Feed-forward activation function

dropout_seed: u64

Seed used by the dropout layers so training runs are reproducible

update_modulation: f64

Amplitude of the transformer’s modulation of the gradient step.

The applied per-coordinate scale is 1 + modulation * tanh(network output), so values in (0, 1) keep the scale strictly positive and the update in the descent half-space.

gradient_checkpointing: bool

Enable gradient checkpointing

attention_optimization: AttentionOptimization

Attention pattern optimization

multi_scale_attention: bool

Multi-scale attention

cross_attention: bool

Cross-attention for multi-task learning

memory_efficient: bool

Memory efficiency mode

Implementations

impl TransformerOptimizerConfig

pub fn validate(&self) -> Result<()>

Validate the configuration, returning a descriptive error for any combination that would otherwise fail (or silently misbehave) later.

Trait Implementations

impl Clone for TransformerOptimizerConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TransformerOptimizerConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TransformerOptimizerConfig

fn default() -> Self

Returns the “default value” for a type.