Struct EnhancedDistributedTrainer 

pub struct EnhancedDistributedTrainer<T: Optimizer + StatefulOptimizer> { /* private fields */ }

Enhanced distributed trainer with modern GPU optimizations

Implementations

impl<T: Optimizer + StatefulOptimizer + Clone> EnhancedDistributedTrainer<T>

pub fn new(config: DistributedConfig, optimizer: T) -> Result<Self>

Create new enhanced distributed trainer

pub fn register_model( &mut self, parameters: HashMap<String, Tensor>, ) -> Result<()>

Register model parameters for distributed training

pub fn train_step( &mut self, gradients: HashMap<String, Tensor>, ) -> Result<TrainingStepResult>

Perform one training step with enhanced distributed optimizations.

The step compresses gradients with the configured codec, reduces them across the multi-node group when one is configured, and decompresses the result. The reduced gradients are retained; because this trainer holds parameter metadata only (see ParameterInfo), the caller — who owns the parameter tensors — applies them with EnhancedDistributedTrainer::apply_reduced_gradients or reads them via EnhancedDistributedTrainer::take_reduced_gradients.

Dynamic batch sizing runs only when device telemetry has been recorded for every device (see EnhancedDistributedTrainer::record_gpu_telemetry); without it there is nothing to base a resize on and the batch sizes are left alone.

pub fn record_gpu_telemetry( &mut self, device_id: usize, sample: GpuTelemetrySample, ) -> Result<()>

Record a real telemetry reading for one device.

This crate cannot read a GPU itself: NVML and ROCm SMI are C libraries and the default build is pure Rust. Rather than inventing plausible numbers, every device metric stays None until the embedder calls this with values it obtained from the platform. Metrics gathered here drive dynamic batch sizing, bottleneck detection and auto-scaling, so a fabricated sample would propagate into real training decisions.

Errors

Fails when device_id is not one of the configured devices or when the sample is out of range (see GpuTelemetrySample).

pub fn take_reduced_gradients(&mut self) -> HashMap<String, Tensor>

Take the gradients produced by the last EnhancedDistributedTrainer::train_step, leaving the trainer empty.

pub fn reduced_gradients(&self) -> &HashMap<String, Tensor>

Borrow the gradients produced by the last EnhancedDistributedTrainer::train_step.

pub fn apply_reduced_gradients( &mut self, parameters: &mut HashMap<String, Tensor>, ) -> Result<usize>

Apply the gradients from the last EnhancedDistributedTrainer::train_step to parameters with this trainer’s optimizer.

Parameters are visited in sorted-name order so every rank performs the same update sequence. Returns the number of parameters updated.

Errors

Fails when a gradient has no matching parameter — a silent skip would leave part of the model un-trained without any signal.

pub fn get_training_stats(&self) -> DistributedTrainingStats

Get comprehensive training statistics.

gpu_utilization and memory_usage are empty when no device telemetry has been recorded; see EnhancedDistributedTrainer::record_gpu_telemetry.

pub fn training_stats_report(&self) -> String

Render the training statistics as a human-readable report.

Prefer this over Self::print_training_stats inside libraries: it returns the text instead of writing to stdout.

pub fn print_training_stats(&self)

Write Self::training_stats_report to stdout.

This is an explicit, caller-initiated escape hatch for binaries and examples; nothing on the training path writes to stdout. Library callers should prefer Self::log_training_stats, which routes the same report through the log facade so the host application controls the sink.

pub fn log_training_stats(&self)

Emit Self::training_stats_report at info level through the log facade.

pub fn checkpoint_due(&self) -> bool

Whether the fault handler considers a checkpoint due at the current step. The caller owns the model state and drives crate::advanced_distributed_features::SmartCheckpointManager.

pub fn optimize_hyperparameters(&mut self) -> Result<T>

Optimize hyperparameters for the current distributed setup.

Errors

Distributed-aware hyperparameter optimization is not implemented. The crate ships crate::hyperparameter_tuning, but wiring it here requires an evaluation callback (a way to run a trial and score it) that this trainer does not have. Rather than returning an unmodified clone of the optimizer while reporting success, this returns TrustformersError describing what is missing whenever auto-tuning is requested.

With config.monitoring.auto_tuning == false the call is a no-op and returns the current optimizer unchanged, which is honest: no optimization was requested and none was performed.