Struct ThreeDParallelismConfig 

pub struct ThreeDParallelismConfig {
    pub dp_size: usize,
    pub tp_size: usize,
    pub pp_size: usize,
    pub num_layers: usize,
    pub micro_batch_size: usize,
    pub memory_strategy: MemoryOptimizationStrategy,
    pub comm_strategy: CommunicationStrategy,
    pub enable_gradient_checkpointing: bool,
    pub enable_mixed_precision: bool,
    pub pipeline_schedule: PipelineSchedule,
    pub max_memory_per_device: f32,
    pub communication_timeout_ms: u64,
}

Configuration for 3D parallelism (Data, Tensor, Pipeline)

Fields

dp_size: usize

Data parallel dimension size

tp_size: usize

Tensor parallel dimension size

pp_size: usize

Pipeline parallel dimension size

num_layers: usize

Total number of layers in the model

micro_batch_size: usize

Micro-batch size for pipeline parallelism

memory_strategy: MemoryOptimizationStrategy

Memory optimization strategy

comm_strategy: CommunicationStrategy

Communication optimization strategy

enable_gradient_checkpointing: bool

Whether to enable gradient checkpointing

enable_mixed_precision: bool

Whether to enable mixed precision training

pipeline_schedule: PipelineSchedule

Pipeline schedule type

max_memory_per_device: f32

Maximum memory usage per device (in GB)

communication_timeout_ms: u64

Communication timeout in milliseconds

Implementations

impl ThreeDParallelismConfig

pub fn validate(&self, world_size: usize) -> TorshResult<()>

Validate configuration against available world size

pub fn layers_per_stage(&self) -> usize

Get number of layers per pipeline stage

pub fn memory_requirements(&self) -> MemoryRequirements

Calculate memory requirements per device

Trait Implementations

impl Clone for ThreeDParallelismConfig

fn clone(&self) -> ThreeDParallelismConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ThreeDParallelismConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ThreeDParallelismConfig

fn default() -> Self

Returns the “default value” for a type.