Module data 

Available on crate feature std only.

Data module.

Modules

causal

Explicit-label causal-LM examples and packed/padded device collation. Causal data collation independent of architecture and tokenizer selection.

dataloaderdataset

Dataloader module.

datasetdataset

Dataset module.

sampler

Committed sample cursors and explicit topology-changing epoch continuation. Dataset-position recovery independent of GPU completion and loader prefetch.

seq2seq

Explicit aligned encoder-decoder examples and paired packed/padded collation. Paired encoder/decoder collation with explicit already-aligned teacher-forcing targets.

token_batch

Token-budget batches retaining committed distributed sample order. Variable-sized batches with exact token cost and recoverable sample cursors.