rumus-distributed 0.1.0

3D parallelism for RUMUS: Tensor Parallelism, Pipeline Parallelism, async collectives
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13

// SPDX-License-Identifier: Apache-2.0 OR MIT
//! `rumus-distributed` — 3D parallelism for RUMUS.
//!
//! Tensor Parallelism (ColumnParallel + RowParallel), Pipeline Parallelism
//! (1F1B micro-batch schedule), and async collective operations.

pub mod collective;
pub mod pipeline;
pub mod tensor_parallel;

pub use collective::{AllReduceHandle, CollectiveBarrier, CommThread};
pub use pipeline::{PipelineExecutor, PipelineStage};
pub use tensor_parallel::{ColumnParallelLinear, RowParallelLinear};