Expand description
rumus-distributed — 3D parallelism for RUMUS.
Tensor Parallelism (ColumnParallel + RowParallel), Pipeline Parallelism (1F1B micro-batch schedule), and async collective operations.
Re-exports§
pub use collective::AllReduceHandle;pub use collective::CollectiveBarrier;pub use collective::CommThread;pub use pipeline::PipelineExecutor;pub use pipeline::PipelineStage;pub use tensor_parallel::ColumnParallelLinear;pub use tensor_parallel::RowParallelLinear;
Modules§
- collective
- Async collective operations: AllReduce via dedicated comm threads.
- pipeline
- Pipeline parallelism: 1F1B micro-batch schedule with per-micro-batch tapes.
- tensor_
parallel - Tensor Parallelism: ColumnParallelLinear and RowParallelLinear.