Module compression 

Gradient compression algorithms for distributed training.

Implements Top-K sparsification (Stich et al. 2018), Random-K sparsification, 1-bit gradient quantization (Seide et al. 2014), and PowerSGD-style low-rank approximation.

Structs

OneBitQuantizer

1-bit gradient quantization (Seide et al. 2014).

PowerSgdConfig

Configuration for PowerSGD low-rank gradient compression.

RandomKCompressor

Random-K gradient sparsification.

TopKCompressor

Top-K gradient sparsification compressor.

TopKConfig

Configuration for Top-K gradient sparsification (Stich et al. 2018).

Functions

low_rank_compress

Compress a gradient matrix G (m × n) into P (m × r) and Q (n × r) such that G ≈ P · Q^T.

low_rank_decompress

Decompress a low-rank gradient approximation back to a full matrix.