Enum ShardingStrategy 

pub enum ShardingStrategy {
    FullShard,
    ShardGradOp,
    NoShard,
    HybridShard {
        intra_node_size: usize,
    },
}

Sharding strategy for FSDP. Mirrors PyTorch’s ShardingStrategy.

• FullShard — shard parameters, gradients, and optimizer state. Minimum memory, maximum communication. Equivalent to ZeRO-3.
• ShardGradOp — keep parameters replicated on every rank (no parameter all-gather in forward), but shard gradients and optimizer state. Equivalent to ZeRO-2. After optimizer.step(), call FSDP::broadcast_updated_params to re-sync the updated param shards back to every rank.
• NoShard — equivalent to DDP: no sharding, allreduce the full gradient. Provided so the FSDP wrapper can be used as a drop-in replacement for DDP during debugging or for single-node experiments.
• HybridShard — shard within a node (intra-node FullShard) and replicate across nodes (inter-node DDP). Uses two SubBackends derived from the global backend to run intra-node all_gather / reduce_scatter and inter-node allreduce independently. CL-327.

CL-372.

Variants

FullShard

Shard parameters + gradients + optimizer state (ZeRO-3 / full FSDP). This is the default and matches the behavior that existed before CL-372.

ShardGradOp

Shard gradients + optimizer state only, keep params replicated (ZeRO-2).

NoShard

No sharding (ZeRO-0 / DDP equivalent). gradients are allreduced.

HybridShard

Hybrid sharding: shard within a node (intra-node FullShard) + replicate across nodes (inter-node DDP). The intra_node_size is the number of ranks per node (i.e., local GPUs). The total world_size must be a multiple of intra_node_size. CL-327.

Fields

intra_node_size: usize

Trait Implementations

impl Clone for ShardingStrategy

fn clone(&self) -> ShardingStrategy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ShardingStrategy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ShardingStrategy

fn default() -> ShardingStrategy

Returns the “default value” for a type.

impl PartialEq for ShardingStrategy

fn eq(&self, other: &ShardingStrategy) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for ShardingStrategy

impl Eq for ShardingStrategy

impl StructuralPartialEq for ShardingStrategy