Struct Attention 

pub struct Attention {
    pub q_proj: Linear,
    pub k_proj: Linear,
    pub v_proj: Linear,
    pub o_proj: Linear,
    pub n_heads: usize,
    pub head_dim: usize,
    pub num_kv_heads: usize,
    pub dropout_prob: f32,
    pub rope_theta: f32,
}

Causal grouped-query self-attention with RoPE (Llama style).

Fields

q_proj: Linear

Query projection [hidden, n_heads * head_dim].

k_proj: Linear

Key projection [hidden, num_kv_heads * head_dim].

v_proj: Linear

Value projection [hidden, num_kv_heads * head_dim].

o_proj: Linear

Output projection [n_heads * head_dim, hidden].

n_heads: usize

Number of query heads.

head_dim: usize

Dimension of each head (at most 128).

num_kv_heads: usize

Number of key/value heads; divides n_heads.

dropout_prob: f32

Attention dropout probability.

rope_theta: f32

RoPE base frequency (rope_theta in Hugging Face configs).

Implementations

impl Attention

pub fn forward( &self, engine: &ComputeEngine, x: &Tensor, pos: usize, cache: Option<&KVCacheLayer>, mask: Option<&UnifiedTensor>, ) -> Tensor

Attends over x ([batch, seq, hidden]) whose first token is at position pos. With cache, the new keys/values are written at pos and the queries attend to the whole cache up to their own position.

pub fn params(&self) -> Vec<Tensor>

Trainable tensors (weights, then any LoRA adapters).

pub fn named_params(&self, prefix: &str) -> HashMap<String, Tensor>

Parameters keyed by Hugging Face-style names under prefix.

pub fn lora_params(&self) -> Vec<Tensor>

Only the LoRA adapter tensors (what LoRA fine-tuning updates).