Struct TensorParallelismConfig 

pub struct TensorParallelismConfig {
    pub tensor_parallel_size: usize,
    pub partitioning_strategy: TensorPartitioningStrategy,
    pub column_parallel: bool,
    pub row_parallel: bool,
    pub communication_pattern: TensorCommunicationPattern,
    pub async_communication: bool,
    pub fusion_threshold_bytes: usize,
    pub gradient_accumulation: bool,
    pub memory_optimization: TensorMemoryOptimization,
    pub mixed_precision: bool,
}

Tensor Parallelism Configuration

Tensor parallelism distributes individual tensors (weights, activations) across multiple devices, enabling the training of models where individual layers are too large to fit on a single device. This is particularly effective for large linear layers and attention mechanisms.

Fields

tensor_parallel_size: usize

Number of devices for tensor parallelism

partitioning_strategy: TensorPartitioningStrategy

Tensor partitioning strategy

column_parallel: bool

Whether to use column parallelism for linear layers

row_parallel: bool

Whether to use row parallelism for linear layers

communication_pattern: TensorCommunicationPattern

Communication pattern for tensor operations

async_communication: bool

Whether to use asynchronous communication

fusion_threshold_bytes: usize

Communication fusion threshold (operations below this size are fused)

gradient_accumulation: bool

Whether to use gradient accumulation across tensor chunks

memory_optimization: TensorMemoryOptimization

Memory optimization level for tensor parallelism

mixed_precision: bool

Whether to use mixed precision for tensor operations

Trait Implementations

impl Clone for TensorParallelismConfig

fn clone(&self) -> TensorParallelismConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TensorParallelismConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TensorParallelismConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TensorParallelismConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TensorParallelismConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.