Struct ProgressiveDistiller 

pub struct ProgressiveDistiller {
    pub layer_weights: Vec<f32>,
    pub temperature: f32,
}

Progressive Layer-wise Distillation

Distills knowledge progressively from intermediate layers of the teacher to corresponding layers of the student. This helps the student learn better intermediate representations, not just final predictions.

Approach

Instead of only matching final logits, progressive distillation also matches intermediate layer outputs (hidden states, attention weights, etc.) between teacher and student at multiple depths.

Example

use entrenar::distill::ProgressiveDistiller;

let distiller = ProgressiveDistiller::new(vec![1.0, 1.0, 2.0], 2.0);

// Match intermediate layers (e.g., layer 3, 6, 9 of teacher to student)
// let loss = distiller.layer_wise_loss(&student_hiddens, &teacher_hiddens);

Fields

layer_weights: Vec<f32>

Weight for each layer’s distillation loss

temperature: f32

Temperature for distillation

Implementations

impl ProgressiveDistiller

pub fn new(layer_weights: Vec<f32>, temperature: f32) -> Self

Create a new progressive distiller

Arguments

• layer_weights - Weight for each layer (will be normalized)
• temperature - Temperature for softening distributions

Panics

Panics if layer_weights is empty or temperature <= 0

pub fn uniform(num_layers: usize, temperature: f32) -> Self

Create progressive distiller with uniform layer weights

pub fn layer_wise_mse_loss( &self, student_hiddens: &[Array2<f32>], teacher_hiddens: &[Array2<f32>], ) -> f32

Compute layer-wise MSE loss between student and teacher hidden states

Arguments

• student_hiddens - Hidden states from student layers [num_layers]
• teacher_hiddens - Hidden states from teacher layers [num_layers]

Returns

Weighted MSE loss across all layers

pub fn layer_wise_cosine_loss( &self, student_hiddens: &[Array2<f32>], teacher_hiddens: &[Array2<f32>], ) -> f32

Compute layer-wise cosine similarity loss

Encourages student representations to have similar direction to teacher, which can be more robust than MSE.

pub fn combined_loss( &self, student_logits: &Array2<f32>, teacher_logits: &Array2<f32>, student_hiddens: &[Array2<f32>], teacher_hiddens: &[Array2<f32>], labels: &[usize], alpha: f32, beta: f32, ) -> f32

Combined progressive distillation loss

Combines final logit distillation with intermediate layer matching.

Arguments

• student_logits - Final logits from student
• teacher_logits - Final logits from teacher
• student_hiddens - Intermediate hidden states from student
• teacher_hiddens - Intermediate hidden states from teacher
• labels - Ground truth labels
• alpha - Weight for distillation vs hard loss
• beta - Weight for hidden state matching vs logit matching

Trait Implementations

impl Clone for ProgressiveDistiller

fn clone(&self) -> ProgressiveDistiller

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ProgressiveDistiller

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.