Module data 

Data Pipeline for Training

Provides Dataset trait and DataLoader for efficient training data loading. Supports SFT (response-masked) and pretraining (next-token) sample formats.

Structs

Batch

Batch of training data

DataLoader

DataLoader for iterating over datasets with batching

TextDataset

Simple in-memory text dataset for LLM training

TrainingExample

One training example with optional response-only loss mask.

Traits

Dataset

Trait for datasets that provide training samples

Functions

build_sft_example

Tokenize one SFT pair into a response-masked example of at most max_len input tokens.