Struct StatefulTokenBatchSampler 

pub struct StatefulTokenBatchSampler { /* private fields */ }

Available on crate feature std only.

Greedy contiguous token-budget batches over the actual distributed order.

This changes batch boundaries only. Encoded lengths must describe the actual examples returned by the immutable source. No tokenization, truncation, length sorting, duplicate tail rows or cross-document supervision is inferred. Different DP coordinates can have different batch counts; the caller must use its training coordinator’s actual participation/accumulation contract.

Implementations

impl StatefulTokenBatchSampler

pub fn new( sampler: StatefulShardSampler, lengths: Vec<usize>, options: TokenBatchOptions, ) -> Result<Self, SamplerError>

Wrap a committed distributed order with exact original-source lengths.

pub fn from_state(state: TokenBatchState) -> Result<Self, SamplerError>

Restore committed samples; all prefetched-but-unconsumed rows are replayed.

pub fn state(&self) -> TokenBatchState

Capture the committed continuation without saving an issued prefetch cursor.

pub fn load_state(&mut self, state: TokenBatchState) -> Result<(), SamplerError>

Replace a matching continuation; reject silently changed lengths/limits.

pub fn reshard( state: &ConsolidatedTokenBatchState, rank: usize, world_size: usize, ) -> Result<Self, SamplerError>

Assign unconsumed examples to a new data topology and repartition batches.

pub fn sampler(&self) -> &StatefulShardSampler

Underlying actual committed/issued order, without mutable aliasing.

pub fn commit(&mut self, examples: usize) -> Result<(), SamplerError>

Commit actual consumed examples, never token slots or prefetched batches.

pub fn rewind_uncommitted(&mut self)

Replay all outstanding prefetch when constructing a fresh loader iterator.

pub fn start_epoch(&mut self, epoch: u64) -> Result<(), SamplerError>

Start the next explicitly selected epoch on the current data topology.

pub fn peek(&self) -> Result<Option<TokenBatchPlan>, SamplerError>

Inspect the next exact batch without advancing either sample cursor.

pub fn next_plan(&mut self) -> Result<Option<TokenBatchPlan>, SamplerError>

Issue the next batch of real indices; no lookahead sample is prematurely issued.

pub fn load_next<I, D: Dataset<I>>( &mut self, dataset: &D, ) -> Result<Option<TokenBatch<I>>, SamplerError>

Available on crate feature dataset only.

Load every actual example before advancing the issued cursor.

pub fn load_batch<B: Backend, I, O, D, F>( &mut self, dataset: &D, batcher: &F, device: &B::Device, ) -> Result<Option<(TokenBatchPlan, O)>, SamplerError>

where D: Dataset<I>, F: Batcher<B, I, O>,

Available on crate feature dataset only.

Batch the loaded actual records on the explicit backend device.