Struct LoraLinear 

pub struct LoraLinear {
    pub frozen_weight: NodeId,
    pub lora_a: NodeId,
    pub lora_b: NodeId,
    pub bias: Option<NodeId>,
    pub scaling: f32,
    pub in_features: usize,
    pub out_features: usize,
    pub rank: usize,
}

A LoRA adapter for a linear layer.

Wraps a frozen weight matrix with trainable low-rank A and B matrices. The effective weight is W + A @ B * scaling where scaling = alpha / rank.

In this crate, weight is stored as [in_features, out_features] (matching LinearLayer), so:

• lora_a has shape [in_features, rank]
• lora_b has shape [rank, out_features]
• Forward: y = x @ W + x @ A @ B * scaling + bias

Fields

frozen_weight: NodeId

Original frozen weight [in_features, out_features].

lora_a: NodeId

Low-rank matrix A [in_features, rank] — initialized with small random values.

lora_b: NodeId

Low-rank matrix B [rank, out_features] — initialized to zeros.

bias: Option<NodeId>

Optional bias (frozen).

scaling: f32

Scaling factor = alpha / rank.

in_features: usize

out_features: usize

rank: usize

Implementations

impl LoraLinear

pub fn new( graph: &mut Graph, in_features: usize, out_features: usize, config: &LoraConfig, ) -> Result<Self, ModelError>

Creates a LoRA adapter from dimensions.

Initializes frozen_weight and bias as graph constants (requires_grad=false), lora_a with small Gaussian-like values (requires_grad=true), lora_b with zeros (requires_grad=true).

pub fn from_linear( graph: &mut Graph, weight_node: NodeId, bias_node: NodeId, in_features: usize, out_features: usize, config: &LoraConfig, ) -> Result<Self, ModelError>

Creates a LoRA adapter from an existing LinearLayer’s weights.

The original weight and bias are frozen (stored as constants). New trainable lora_a and lora_b matrices are created.

pub fn with_bias( self, graph: &mut Graph, bias_tensor: Tensor, ) -> Result<Self, ModelError>

Creates a LoRA adapter with a frozen bias term.

pub fn forward( &self, graph: &mut Graph, input: NodeId, ) -> Result<NodeId, ModelError>

Forward pass: y = x @ W + x @ A @ B * scaling + bias.

Gradients flow through A and B but not through the frozen weight.

pub fn trainable_params(&self) -> Vec<NodeId>

Returns the trainable parameter NodeIds (only lora_a and lora_b).

pub fn merge(&self, graph: &Graph) -> Result<Tensor, ModelError>

Merges LoRA weights into the frozen weight, returning the effective weight tensor.

W_eff = W + A @ B * scaling

Useful for inference after training to avoid the LoRA overhead.

Trait Implementations

impl Clone for LoraLinear

fn clone(&self) -> LoraLinear

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LoraLinear

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.