Expand description
Tensor Parallelism: ColumnParallelLinear and RowParallelLinear.
Structsยง
- Column
Parallel Linear - Linear layer with weight sharded along columns (N dimension).
- RowParallel
Linear - Linear layer with weight sharded along rows (K dimension).