Module parallel 

Multi-GPU Parallelism Module

This module provides support for distributing model inference across multiple GPUs using various parallelism strategies:

• Tensor Parallelism: Split tensor operations across GPUs
• Pipeline Parallelism: Split model layers across GPUs
• Data Parallelism: Process different batches on different GPUs

Architecture

┌─────────────────────────────────────────────────────────┐
│                    DeviceManager                         │
│  - Device discovery and capability detection             │
│  - Resource allocation and monitoring                    │
└─────────────────────────────────────────────────────────┘
                             │
         ┌───────────────────┼───────────────────┐
         ▼                   ▼                   ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ TensorParallel  │ │ PipelineParallel│ │  DataParallel   │
│   Executor      │ │   Executor      │ │   Executor      │
└─────────────────┘ └─────────────────┘ └─────────────────┘

Re-exports

pub use config::LayerDistribution;

pub use config::LayerRange;

pub use config::ParallelConfig;

pub use config::ParallelismType;

pub use device::global_device_manager;

pub use device::DeviceCapability;

pub use device::DeviceInfo;

pub use device::DeviceManager;

pub use executor::ParallelExecutor;

pub use executor::ParallelExecutorFactory;

pub use executor::ParallelStrategySelector;

pub use tensor_parallel::LayerParallelType;

pub use tensor_parallel::TensorParallelConfig;

pub use tensor_parallel::TensorParallelGroup;

pub use tensor_parallel::TransformerParallelMapping;

pub use tensor_parallel::WeightShard;

Modules

config

Parallel Configuration

device

Device Management

executor

Parallel Executor

tensor_parallel

Tensor Parallelism