Module distributed_optimizer 

Distributed gradient optimizer — coordinates gradient aggregation across multiple distributed training workers with staleness handling, compression-friendly interfaces, and fault-tolerant worker management.

Overview

DistributedOptimizer is the central coordinator. Workers register, submit gradient updates for named layers, and the optimizer aggregates them according to the chosen AggregationStrategy.

Four aggregation strategies are supported:

• Synchronous — classical barrier-based: every active worker must submit before aggregation proceeds.
• Asynchronous — accepts updates within a configurable staleness window; stale updates are rejected at submission time.
• FederatedAverage — aggregates as soon as at least rounds updates have been collected for a layer.
• GossipAverage — aggregates as soon as at least fanout updates are available, mimicking gossip protocol averaging.

Structs

AggregatedGradient

The result of aggregating gradient updates for a single layer.

DistOptimizerStats

A point-in-time snapshot of optimizer statistics.

DistributedOptimizer

Coordinator for distributed gradient aggregation across multiple workers.

GradientUpdate

A single gradient update submitted by one worker for one named layer.

WorkerId

Opaque identifier for a training worker.

WorkerState

Liveness and bookkeeping state for a single registered worker.

Enums

AggregationStrategy

Determines how and when pending updates are aggregated.

OptimizerDistError

Errors that can arise from DistributedOptimizer operations.