Trait Dataset 

pub trait Dataset: Send + Sync {
    // Required methods
    fn len(&self) -> usize;
    fn get(&self, idx: usize) -> (Vec<u32>, Vec<u32>);

    // Provided methods
    fn get_loss_mask(&self, idx: usize) -> Option<Vec<u8>> { ... }
    fn is_empty(&self) -> bool { ... }
}

Trait for datasets that provide training samples

Required Methods

fn len(&self) -> usize

Number of samples in the dataset

fn get(&self, idx: usize) -> (Vec<u32>, Vec<u32>)

Get a single sample by index Returns (input_ids, target_ids) as token vectors

Provided Methods

fn get_loss_mask(&self, idx: usize) -> Option<Vec<u8>>

Optional per-token loss mask aligned with targets (1 = include, 0 = ignore)

fn is_empty(&self) -> bool

Check if dataset is empty

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl Dataset for TextDataset