Expand description
ZeRO Optimizer (Zero Redundancy Optimizer)
Implements memory-efficient distributed training:
- ZeRO Stage 1: Optimizer state partitioning
- ZeRO Stage 2: Gradient partitioning
- ZeRO Stage 3: Parameter partitioning
- ZeRO-Offload: CPU/NVMe offloading
- Communication optimization
Structs§
- Parameter
Partition - Parameter partition information
- ZeRo
Config - ZeRO configuration
- ZeRo
Optimizer - ZeRO optimizer state
- ZeRo
Stats - ZeRO statistics
Enums§
- ZeRo
Stage - ZeRO stage configuration