Struct CompressedAllReduce 

pub struct CompressedAllReduce { /* private fields */ }

Distributed gradient aggregator with compression support

Implementations

impl CompressedAllReduce

pub fn new(compression_method: CompressionMethod, world_size: usize) -> Self

pub fn all_reduce( &mut self, gradients: &HashMap<String, Tensor>, ) -> Result<HashMap<String, Tensor>>

Compresses, aggregates and averages gradients across the process group.

Aggregation semantics

No transport is wired up in this crate, so there is exactly one honest thing this can do:

• world_size == 1 — the local rank is the group. The gradient is compressed and decompressed (so the caller sees the real compression error) and returned.
• world_size > 1 with peer contributions supplied — see CompressedAllReduce::all_reduce_with_peers, which performs the real sum.
• world_size > 1 with no peers — an error. Scaling the local gradient by world_size (what this used to do) is not an approximation of a sum across workers; it is numerically worse than doing nothing.

Errors

Returns an error when world_size > 1, because no communicator is configured.

pub fn all_reduce_with_peers( &mut self, gradients: &HashMap<String, Tensor>, peers: &[HashMap<String, CompressedGradient>], ) -> Result<HashMap<String, Tensor>>

Performs a genuine compressed all-reduce given the peers’ compressed gradients.

peers holds one map per other rank. Each parameter’s contributions are summed in the dense domain and divided by the number of contributing ranks, so the result is the true average of the compressed gradients — not a rescaling of the local one.

Errors

Returns an error when a peer supplies a gradient of a different length, or when the number of contributions does not match world_size.

pub fn compress_for_transmission( &mut self, gradients: &HashMap<String, Tensor>, ) -> Result<HashMap<String, CompressedGradient>>

Compresses gradients for transmission to the other ranks.

Errors

Returns an error when a gradient cannot be read.

Trait Implementations

impl Debug for CompressedAllReduce

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.