Struct EmbeddingGatherParams 

pub struct EmbeddingGatherParams {
    pub embed_dim: usize,
    pub group_size: usize,
    pub bits: u8,
    pub n_tokens: usize,
}

Parameters for quantized embedding gather.

Fields

embed_dim: usize

Embedding dimension (number of float values per token).

group_size: usize

Number of elements per quantization group (typically 64).

bits: u8

Quantization bit width: 4 or 6.

n_tokens: usize

Number of tokens to gather.