Struct KnowledgeDistillation 

pub struct KnowledgeDistillation { /* private fields */ }

Knowledge Distillation (Hinton et al., 2015).

Transfers knowledge from a large teacher model to a smaller student model by training the student to match the teacher’s soft predictions.

Variants

• Standard: Match soft logits with temperature scaling
• Feature: Match intermediate layer representations
• Attention: Match attention patterns
• Self: Use deeper layers to teach shallower layers

Reference

• Hinton, G., et al. (2015). Distilling the Knowledge in a Neural Network.

Implementations

impl KnowledgeDistillation

pub fn new(temperature: f32, alpha: f32) -> Self

Create knowledge distillation with temperature and mixing weight.

Arguments

• temperature - Softmax temperature (higher = softer targets)
• alpha - Weight of distillation loss vs task loss (0-1)

pub fn distillation_loss( &self, student_logits: &[f32], teacher_logits: &[f32], ) -> f32

Compute soft cross-entropy loss between teacher and student logits.

pub fn combined_loss( &self, student_logits: &[f32], teacher_logits: &[f32], task_loss: f32, ) -> f32

Compute combined loss: alpha * distill_loss + (1-alpha) * task_loss.

pub fn temperature(&self) -> f32

pub fn alpha(&self) -> f32

Trait Implementations

impl Clone for KnowledgeDistillation

fn clone(&self) -> KnowledgeDistillation

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KnowledgeDistillation

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.