Enum ParallelismMode 

pub enum ParallelismMode {
    TensorParallel,
    ReplicatedData,
}

Parallelism strategy for KV cache across workers.

This determines how KV blocks are distributed and transferred across multiple workers in a distributed inference setup.

Variants

TensorParallel

Tensor parallel: each worker has a shard of each KV block.

This is the standard approach for tensor-parallel inference where attention heads are split across workers. Each worker stores and transfers only its portion of each KV block.

All workers have G1, G2, and G3 tiers. Operations execute on all workers simultaneously (SPMD).

ReplicatedData

Replicated data: all workers have full KV blocks (MLA scenario).

In MLA (Multi-head Latent Attention) architectures, KV blocks are replicated rather than sharded. Only rank 0 has G2/G3 storage; data is broadcast to other ranks after loading to G1.

This reduces storage requirements on non-rank-0 workers and is suitable when the model’s KV representation is the same across all attention heads.

Trait Implementations

impl Clone for ParallelismMode

fn clone(&self) -> ParallelismMode

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for ParallelismMode

impl Debug for ParallelismMode

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ParallelismMode

fn default() -> ParallelismMode

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for ParallelismMode

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for ParallelismMode

impl PartialEq for ParallelismMode

fn eq(&self, other: &ParallelismMode) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for ParallelismMode

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for ParallelismMode