Struct MultiHeadAttentionWithLoRA 

pub struct MultiHeadAttentionWithLoRA {
    pub config: TransformerConfig,
    pub q_proj: LoRAProjection,
    pub k_proj: LoRAProjection,
    pub v_proj: LoRAProjection,
    pub o_proj: LoRAProjection,
}

Multi-head attention with deep LoRA injection

LoRA adapters are applied to Q, K, V, O projections during forward pass

Fields

config: TransformerConfig

Configuration

q_proj: LoRAProjection

Query projection with LoRA

k_proj: LoRAProjection

Key projection with LoRA

v_proj: LoRAProjection

Value projection with LoRA

o_proj: LoRAProjection

Output projection with LoRA

Implementations

impl MultiHeadAttentionWithLoRA

pub fn from_attention( attn: &MultiHeadAttention, rank: usize, alpha: f32, ) -> Self

Create LoRA-enabled attention from existing attention weights

Arguments

• attn - Base MultiHeadAttention with pretrained weights
• rank - LoRA rank
• alpha - LoRA alpha scaling factor

pub fn forward(&self, x: &Tensor, seq_len: usize) -> Tensor

Forward pass with deep LoRA injection

LoRA is applied to all Q, K, V, O projections

pub fn lora_params(&self) -> Vec<&Tensor>

Get all trainable LoRA parameters

pub fn lora_params_mut(&mut self) -> Vec<&mut Tensor>

Get all trainable LoRA parameters as mutable references

pub fn lora_param_count(&self) -> usize

Count total LoRA parameters