rusttorch-data 0.4.0

Typed datasets, samplers, batching, and loading for RustTorch
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
# rusttorch-data

Build training and inference input pipelines from Rust datasets and iterators.
Select rows, shuffle each epoch, transform samples, assemble typed batches,
and prefetch with bounded worker threads. Long-running jobs can save a loader
checkpoint and resume from the next unconsumed batch.

Applications using `rusttorch` can import the same API from `rusttorch::data`.
Use this package directly when you only need the data layer.

## Installation

```toml
[dependencies]
rusttorch-data = { version = "0.4", features = ["download-libtorch"] }
rusttorch-core = "0.4"
```

`rusttorch-core` supplies `Tensor`, `Kind` and `Device` for tensor pipelines.
Use the same release series for both packages.

## Features

| Feature | Use it for |
|---|---|
| No default features | Link to a runtime you provide. |
| `download-libtorch` | Download the compatible LibTorch 2.13.0 runtime through `tch`. |
| `doc-only` | Build API documentation and type-check without a native runtime. Executables need a real runtime. |

## Native runtime

RustTorch tensors use LibTorch through `tch` 0.26.0. An executable needs
LibTorch 2.13.0 and its shared libraries on the platform's library search path.
Instead of downloading it, set `LIBTORCH=/absolute/path/to/libtorch`, or use
`LIBTORCH_USE_PYTORCH=1` with an installed Python `torch` 2.13.0. The Python
installation is an optional way to provide these native libraries; the loader
and application code are Rust.

## Example: batch features and labels

`TensorDataset` keeps matching feature and label tensors aligned. Its first
axis indexes samples; `DefaultCollator` stacks the selected rows into a batch.
Use this path for small datasets already held in memory.

```rust
use rusttorch_core::Tensor;
use rusttorch_data::{DataLoader, TensorDataset};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let features = Tensor::from_slice(&[1_f32, 2., 3., 4., 5., 6.]).reshape([3, 2]);
    let labels = Tensor::from_slice(&[0_i64, 1, 0]);
    let dataset = TensorDataset::new(vec![features, labels])?;
    let mut loader = DataLoader::builder(dataset)
        .shuffle(42)?
        .batch_size(2)
        .build()?;

    assert_eq!(loader.len(), Some(2));
    for batch in loader.iter() {
        let batch = batch?;
        let inputs = &batch[0];
        let targets = &batch[1];
        assert_eq!(inputs.size()[0], targets.size()[0]);
        assert_eq!(inputs.size()[1], 2);
        // Pass inputs to your model and targets to its loss function.
    }
    Ok(())
}
```

The last batch contains one sample. Set `.drop_last(true)` when every visible
batch must have the configured size. Use `.collate(VecCollate)` to keep Rust
values as a `Vec`, or `.collate(FnCollate::new(...))` for your own padding or
batch representation. `.without_batching()` converts one sample at a time.

## Choose a loading path

| Input or use case | API |
|---|---|
| Inspect a dataset on the calling thread | `DataLoader::new(&dataset, sampler, batch_size, drop_last)` |
| Reuse a map dataset over multiple epochs | `DataLoader::builder(dataset)` and `loader.iter()` |
| Decode indexed files concurrently | Implement `Dataset`, then select `.workers(count)` |
| Batch a serial iterator | `batches(iterator, batch_size, drop_last)` |
| Read independently owned stream shards concurrently | Implement `WorkerSourceFactory`; use `StreamDataLoaderBuilder` |
| Select a subset or training/validation split | `Subset` or `random_split` |
| Combine map datasets or iterator streams | `ConcatDataset`, `StackDataset`, or `chain_datasets` |
| Address class imbalance | `WeightedRandomSampler` |
| Partition one dataset across training ranks | `DistributedSampler` |

The borrowed `DataLoader::new` returns an iterator directly: iterate over it
without `build()` or `iter()`. It keeps your dataset available to inspect after
loading and runs each fetch on the calling thread, which is useful for debugging.

A complete borrowed pass keeps the dataset available afterward. It does not
stack samples automatically, so the output here is `Vec<i64>`:

```rust
use rusttorch_data::{DataLoader, Dataset};
use std::convert::Infallible;
struct Rows([i64; 3]);
impl Dataset for Rows {
    type Sample = i64;
    type Error = Infallible;
    fn len(&self) -> usize {
        self.0.len()
    }
    fn get(&self, i: usize) -> Result<i64, Infallible> {
        Ok(self.0[i])
    }
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let dataset = Rows([5, 8, 13]);
    let loader = DataLoader::new(&dataset, 0..dataset.len(), 2, false)?;
    assert_eq!(
        loader.collect::<Result<Vec<_>, _>>()?,
        [vec![5, 8], vec![13]]
    );
    assert_eq!(dataset.get(0)?, 5); // The dataset is still available.
    Ok(())
}
```

For a custom `Dataset`, implement `len()` and `get(index)`. Samples and errors
keep their concrete Rust types. `get_batch(indices)` is optional: override it
when one database query or decode operation can fetch an entire index batch.

[`examples/loader.rs`](examples/loader.rs) is a complete, runnable example of:

- shuffled tensor batches with four workers and automatic pinning;
- distributed sampling with `set_epoch(epoch)` before each epoch; and
- lazy stream shards merged in global order with one final short batch.

In a repository checkout with the native runtime configured, run:

```sh
cargo run -p rusttorch-data --example loader --locked
```

Distributed ranks must agree on dataset length, replica count, seed and epoch.
Set `.rank(rank)` on the loader as well as selecting the sampler's rank.
The sampler partitions input indices; gradient synchronization belongs to
training. Sampler `drop_last` controls division across ranks; loader
`drop_last` separately controls the final local batch.

## Train with typed feature/label batches

When building a complete application, use the facade dependency instead of
importing the data and core crates separately:

```toml
[dependencies]
rusttorch = { version = "0.4", features = ["download-libtorch"] }
```

The default collator turns `(Tensor, i64)` samples into `(Tensor, Tensor)`
batches. Feature rows are stacked, and labels become an `Int64` tensor suitable
for classification loss. The dataset below stores ordinary Rust values and
creates each tensor when fetched, so workers can share it safely.

```rust
use rusttorch::{
    DeviceSpec, Tensor,
    data::{DataLoader, Dataset},
    nn::{Sequential, functional},
    optim::Adam,
};
use std::convert::Infallible;

struct Examples(Vec<([f32; 2], i64)>);
impl Dataset for Examples {
    type Sample = (Tensor, i64);
    type Error = Infallible;
    fn len(&self) -> usize {
        self.0.len()
    }
    fn get(&self, index: usize) -> Result<Self::Sample, Infallible> {
        let (features, label) = &self.0[index];
        Ok((Tensor::from_slice(features), *label))
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let examples = Examples(vec![
        ([-1., 0.], 0),
        ([1., 0.], 1),
        ([-2., 1.], 0),
        ([2., 1.], 1),
    ]);
    let mut loader = DataLoader::builder(examples)
        .batch_size(2)
        .shuffle(42)?
        .workers(2)
        .prefetch_factor(2)
        .build()?;
    let model = Sequential::builder().linear(2, 2).build(DeviceSpec::Cpu)?;
    let mut optimizer = Adam::builder()
        .learning_rate(0.01)
        .build(model.var_store())?;

    for epoch in 0..3 {
        loader.set_epoch(epoch);
        for batch in loader.iter() {
            let (features, labels) = batch?;
            assert_eq!(features.size(), [2, 2]);
            assert_eq!(labels.size(), [2]);
            let logits = model.forward_t(&features, true)?;
            let loss = functional::cross_entropy(&logits, &labels)?;
            optimizer.backward_step(&loss)?;
        }
    }
    Ok(())
}
```

Calling `iter()` again repeats the selected epoch. Call `set_epoch(epoch)` to
select that epoch's deterministic shuffle before its first batch. `.shuffle(42)`
sets the sampler seed; `.seed(...)` separately controls task/worker randomness.
The example keeps the model and inputs on CPU. When training on an accelerator,
move both feature and target tensors to the model's device before the loss.

## Pad sequences or return individual samples

Custom collation runs after per-sample transforms. Use it to pad variable-length
sequences and retain the lengths needed by the model. The following program
returns token IDs shaped `[batch, longest_sequence]` and an `Int64` length vector.
Zero is the chosen padding token; use a different value if your vocabulary
assigns zero to an ordinary token.

```rust
use rusttorch_core::Tensor;
use rusttorch_data::{DataLoader, Dataset, FnCollate};
use std::convert::Infallible;
struct Sentences(Vec<Vec<i64>>);
impl Dataset for Sentences {
    type Sample = Vec<i64>;
    type Error = Infallible;
    fn len(&self) -> usize {
        self.0.len()
    }
    fn get(&self, index: usize) -> Result<Self::Sample, Infallible> {
        Ok(self.0[index].clone())
    }
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let padding = FnCollate::new(|rows: Vec<Vec<i64>>| {
        let width = rows.iter().map(Vec::len).max().unwrap_or(0);
        let lengths: Vec<_> = rows.iter().map(|row| row.len() as i64).collect();
        let count = rows.len();
        let mut values = Vec::new();
        for mut row in rows {
            row.resize(width, 0);
            values.extend(row);
        }
        Ok::<_, rusttorch_core::RustTorchError>((
            Tensor::f_from_slice(&values)?.f_reshape([count as i64, width as i64])?,
            Tensor::f_from_slice(&lengths)?,
        ))
    });
    let mut loader = DataLoader::builder(Sentences(vec![vec![1, 2], vec![3, 4, 5]]))
        .batch_size(2)
        .collate(padding)
        .build()?;
    let (tokens, lengths) = loader.iter().next().unwrap()?;
    assert_eq!(
        Vec::<Vec<i64>>::try_from(&tokens)?,
        [vec![1, 2, 0], vec![3, 4, 5]]
    );
    assert_eq!(Vec::<i64>::try_from(&lengths)?, [2, 3]);
    Ok(())
}
```

For inference on one record at a time, or when each record is already a batch,
use `without_batching()`. It converts samples without inserting a batch axis;
do not combine it with `batch_size` or `drop_last`.

```rust
use rusttorch_core::Tensor;
use rusttorch_data::{DataLoader, TensorDataset};
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let features = Tensor::from_slice(&[1_f32, 2., 3., 4.]).reshape([2, 2]);
    let mut loader = DataLoader::builder(TensorDataset::new(vec![features])?)
        .without_batching()
        .build()?;
    for sample in loader.iter() {
        assert_eq!(sample?[0].size(), [2]);
    }
    Ok(())
}
```

## Prefetch, transform, and transfer

Owned map loaders default to the calling thread. `.workers(4)` shares the map
dataset through `Arc`, so the dataset must be `Send + Sync`; samples must be
`Send`. A dataset can return newly decoded tensors without sharing tensor
storage between threads. Serial loading accepts datasets that cannot meet
those bounds, including `TensorDataset`.

Use `.transform(...)` to transform each sample before collation, or
`.transform_factory(...)` to create a separate decoder or transform for each
worker. `TaskContext` provides deterministic randomness derived from the
loader seed, epoch, rank and logical sample identity. `.worker_init(...)`
initializes per-worker resources once per pool.

`.prefetch_factor(2)` permits at most `workers * 2` outstanding map batches or
stream records. Results preserve sampler/global sequence order by default;
`.ordered(false)` delivers completion order. For streams, each worker owns a
separate iterator. Ordered records need unique, contiguous global `SequenceId`
values starting at zero. Each record also carries a stable `LogicalSampleId`
for task randomness. Stream collation and `drop_last` apply globally after
merging, so uneven shards do not each lose their own tail.

For a stream that emits records out of order, the next global ID must remain
reachable within the prefetch window. Exhausting the window with higher IDs
reports a protocol error. Increase the factor or change the sharding strategy
when a legitimate source needs a wider window.

`.prefetch_bytes(NonZeroUsize)` additionally limits the logical payload held in
prefetched transformed results. Map budgets must fit at least one complete
transformed batch; stream budgets must fit one record. The final transformed
type implements `MemoryFootprint`. Byte-bounded ordered streams require
strictly increasing IDs within each shard. This budget excludes the current
collation batch, callback allocations, tensor aliasing and allocator metadata;
it is not a process memory limit. Build also rejects configurations whose
checked aggregate queue/bookkeeping allocation exceeds 64 MiB.

`.pin_memory()` pins each collated batch on available CUDA device zero.
CPU-only and MPS-only runtimes preserve the batch and report
`PinMemoryStatus::DisabledNoAccelerator`. `.pin_memory_for(Device::Cuda(index))`
selects a specific available CUDA device; unsupported selections fail before
callbacks run. Pinning prepares host memory for transfers; it does not move
batches to the model's device.

`.timeout(duration)` limits each blocking `next()` call with positive workers.
Callbacks can observe deadlines and cancellation through `WorkerContext`.
Dropping an iterator cancels its generation and waits for active work to stop.
With `.persistent_workers(true)`, threads and transforms remain alive for the
next iterator; dropping the loader shuts down and joins the pool. Rust cannot
interrupt a blocking decoder or foreign call that ignores cancellation, so
cleanup waits for that call to return.

## Read lazy stream shards

A worker source factory opens disjoint records for each worker. The loader does
not partition your data automatically: returning the full input from every
worker duplicates records. Use the worker ID and worker count to select a shard.
This program keeps only a lazy range cursor per shard and merges five records
into `[0, 1]`, `[2, 3]`, and one global tail `[4]`.

```rust
use rusttorch_data::{
    LogicalSampleId, SequenceId, StreamDataLoaderBuilder, VecCollate, WorkerContext, WorkerRecord,
    WorkerSourceFactory,
};
use std::{convert::Infallible, iter::StepBy, ops::Range};
struct Rows(usize);
struct Shard(StepBy<Range<usize>>);
impl Iterator for Shard {
    type Item = Result<WorkerRecord<usize>, Infallible>;
    fn next(&mut self) -> Option<Self::Item> {
        self.0.next().map(|row| {
            Ok(WorkerRecord {
                sequence: Some(SequenceId::new(row as u64)),
                logical_id: LogicalSampleId::new(row as u64),
                sample: row,
            })
        })
    }
}
impl WorkerSourceFactory for Rows {
    type Sample = usize;
    type Error = Infallible;
    type Source = Shard;
    fn create(&self, worker: WorkerContext) -> Result<Shard, Infallible> {
        Ok(Shard(
            (worker.info.id..self.0).step_by(worker.info.num_workers),
        ))
    }
    fn exact_len(&self) -> Option<usize> {
        Some(self.0)
    }
}
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut loader = StreamDataLoaderBuilder::new(Rows(5))
        .workers(2)
        .prefetch_factor(2)
        .batch_size(2)
        .collate(VecCollate)
        .build()?;
    let batches = loader.iter().collect::<Result<Vec<_>, _>>()?;
    assert_eq!(batches, [vec![0, 1], vec![2, 3], vec![4]]);
    Ok(())
}
```

For an existing serial iterator, `batches(iterator, size, drop_last)` is the
smaller entry point. Use worker shards when a source can be opened independently
and preparation benefits from concurrency. A source without an exact length
can leave `exact_len()` at its default `None`.

## Checkpoint and resume a training input pipeline

[`examples/checkpoint.rs`](examples/checkpoint.rs) saves an ordered, prefetched
map loader after one batch, round-trips its state through JSON, and verifies
that a recreated loader returns exactly the same remaining samples:

```sh
cargo run -p rusttorch-data --example checkpoint --locked
```

Applications choose the storage format. The complete example below uses JSON,
so add `serde_json = "1"` alongside the data dependencies. The library uses
serde's format-neutral traits in production.

```rust
use rusttorch_data::{DataLoader, Dataset, ReplaySafeDataset, ReplaySafeMap, VecCollate};
use std::convert::Infallible;
struct Rows([i64; 5]);
impl Dataset for Rows {
    type Sample = i64;
    type Error = Infallible;
    fn len(&self) -> usize {
        self.0.len()
    }
    fn get(&self, index: usize) -> Result<i64, Infallible> {
        Ok(self.0[index])
    }
}
impl ReplaySafeDataset for Rows {}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let build = || {
        DataLoader::builder(ReplaySafeMap::new(Rows([2, 3, 5, 7, 11])))
            .batch_size(2)
            .workers(2)
            .prefetch_factor(2)
            .collate(VecCollate)
            .dataset_identity("prime-rows-v1".to_owned())
    };
    let mut loader = build().build()?;
    let mut iteration = loader.iter();
    assert_eq!(iteration.next().unwrap()?, [2, 3]);
    let saved = serde_json::to_string(&iteration.checkpoint()?)?;
    let uninterrupted = iteration.collect::<Result<Vec<_>, _>>()?;

    let mut restored = build().resume_from(serde_json::from_str(&saved)?).build()?;
    let remaining = restored.iter().collect::<Result<Vec<_>, _>>()?;
    assert_eq!(remaining, [vec![5, 7], vec![11]]);
    assert_eq!(remaining, uninterrupted);
    Ok(())
}
```

For an immutable map dataset, implement `ReplaySafeDataset`, wrap it in
`ReplaySafeMap`, and set `.dataset_identity("contents-version".to_owned())`.
Use a content/version identity that changes when the dataset changes.
`TensorDataset::into_replay_safe()` creates isolated copies for serial replay.
Stateful serial datasets use `TransactionalMap` and `TransactionalCheckpoint`.
The identity transform and built-in collators already support checkpoints;
custom components must supply the corresponding state contracts.

Call `iteration.checkpoint()?` initially or after a successful visible batch.
Save the returned `LoaderState` and resume with a matching builder's
`.resume_from(state).build()?`. For a complete training restart, save model
weights, optimizer state and training counters at the same logical step.
The loader checkpoint covers the input pipeline only.

| Loading mode | Exact checkpoint support | Component requirements |
|---|---|---|
| Borrowed map loader or `batches` | Unavailable | Use an owned loader for checkpointing. |
| Serial owned map | Available | Replay-safe or transactional dataset; checkpointable sampler, transform and collator/converter. |
| Ordered map workers | Available | Replay-safe dataset; checkpointable sampler, worker transforms and collator/converter. |
| Ordered stream workers | Available | `CheckpointSourceFactory`, `CheckpointableSource`, checkpointable worker transforms and collator. |
| Completion-order workers | Unavailable | Select ordered delivery for exact replay. |

Exact worker modes additionally require no byte budget, no custom worker
initializer, no persistent workers and no timeout. These configurations are
rejected instead of promising inexact replay. Stateful transforms can opt in
with `.checkpoint_transactional()`; explicitly stateless map transforms use
`.checkpoint_stateless()`. Stream transforms use `WorkerCheckpoint`,
`StatelessWorker`, or `TransactionalWorker`.

For streams, enable `.checkpointable("source-contents-v1")`, call
`iteration.checkpoint()`, then recreate with `.resume("source-contents-v1", state)`.
A source stores an owned serde cursor, validates it without mutation, and
restores it through `CheckpointableSource`. The factory supplies a versioned
`CHECKPOINT_KIND`. Ordered IDs must increase strictly within each shard;
`error_sequence` identifies a failed read's reproducible global position.
Sources retaining `WorkerContext` must replace it in `set_run_context` before
resumed reads. See the [source contract](https://docs.rs/rusttorch-data/latest/rusttorch_data/trait.CheckpointableSource.html).

### Complete stream cursor and JSON restore

A stream reader must save every piece of state affecting its next record.
This example stores each shard's local cursor, validates it without mutation,
and replaces its cancellation context when reads resume. The factory kind
versions the cursor format; the content identity versions the actual rows.
Use the data/core dependencies above plus `serde_json = "1"`.

```rust
use rusttorch_data::{
    CheckpointSourceFactory, CheckpointableSource, LogicalSampleId, SequenceId,
    StreamDataLoaderBuilder, StreamLoaderState, VecCollate, WorkerContext, WorkerRecord,
    WorkerSourceFactory,
};
use std::io;

const ROWS: usize = 11;
struct Rows;
struct Shard {
    cursor: usize,
    context: WorkerContext,
}
impl Shard {
    fn len(&self) -> usize {
        ROWS.saturating_sub(self.context.info.id)
            .div_ceil(self.context.info.num_workers)
    }
}
impl Iterator for Shard {
    type Item = Result<WorkerRecord<usize>, io::Error>;
    fn next(&mut self) -> Option<Self::Item> {
        if self.context.cancellation.is_cancelled() || self.cursor == self.len() {
            return None;
        }
        let row = self.context.info.id + self.cursor * self.context.info.num_workers;
        self.cursor += 1;
        Some(Ok(WorkerRecord {
            sequence: Some(SequenceId::new(row as u64)),
            logical_id: LogicalSampleId::new(row as u64),
            sample: row,
        }))
    }
}
impl WorkerSourceFactory for Rows {
    type Sample = usize;
    type Error = io::Error;
    type Source = Shard;
    fn create(&self, context: WorkerContext) -> Result<Shard, io::Error> {
        Ok(Shard { cursor: 0, context })
    }
    fn exact_len(&self) -> Option<usize> {
        Some(ROWS)
    }
}
impl CheckpointSourceFactory for Rows {
    const CHECKPOINT_KIND: &'static str = "example.modulo-rows.v1";
}
impl CheckpointableSource for Shard {
    type Sample = usize;
    type Error = io::Error;
    type State = usize;
    fn snapshot(&self) -> usize {
        self.cursor
    }
    fn validate_snapshot(&self, cursor: &usize) -> Result<(), io::Error> {
        if *cursor > self.len() {
            return Err(io::Error::new(
                io::ErrorKind::InvalidData,
                "invalid shard cursor",
            ));
        }
        Ok(())
    }
    fn restore_validated(&mut self, cursor: &usize) {
        self.cursor = *cursor;
    }
    fn set_run_context(&mut self, context: WorkerContext) {
        self.context = context;
    }
    fn error_sequence(&self, _: &io::Error) -> SequenceId {
        unreachable!("this in-memory source never returns a read error")
    }
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let build = || {
        StreamDataLoaderBuilder::new(Rows)
            .workers(3)
            .prefetch_factor(2)
            .batch_size(4)
            .collate(VecCollate)
    };
    let mut loader = build().checkpointable("rows-0-through-10.v1").build()?;
    let mut iteration = loader.iter();
    assert_eq!(iteration.next().unwrap()?, [0, 1, 2, 3]);
    let saved = serde_json::to_string(&iteration.checkpoint()?)?;
    let uninterrupted = iteration.collect::<Result<Vec<_>, _>>()?;

    let state: StreamLoaderState<usize> = serde_json::from_str(&saved)?;
    let mut restored = build().resume("rows-0-through-10.v1", state).build()?;
    let remaining = restored.iter().collect::<Result<Vec<_>, _>>()?;
    assert_eq!(remaining, [vec![4, 5, 6, 7], vec![8, 9, 10]]);
    assert_eq!(remaining, uninterrupted);
    Ok(())
}
```

A real file decoder may also need a byte offset, decoder buffer state and
version information. It must report a stable sequence position for read
failures through `error_sequence`. The example only produces successful reads;
its `io::Error` type is used to reject invalid cursor state.

Checkpoints save component state, not decoded samples. Prefetched work is
rolled back to the next consumer-visible boundary and replayed; the original
iterator can continue after checkpointing. Map checkpoints require an initial
or successful batch boundary before any iteration error or observed exhaustion.
Exact streams can also replay a source/transform error from their last successful
boundary; collation/pinning errors, observed exhaustion and hidden dropped tails
cannot be checkpointed. Resume checks the
schema, identity and configuration before restoring component state. Propagate
storage and validation errors rather than silently restarting at the beginning.
For crash recovery, the application owns atomic storage and retention.

## Measure your workload

The dependency-free benchmark checks output checksums and reports two warm-up
runs, seven raw timings, a mean and sample standard deviation for each case.
It covers borrowed/owned serial overhead, worker counts, ordering, prefetch
factors, byte bounds, transforms, collation, automatic pinning, and checkpoint
barriers paired with equivalent work without a barrier.

```sh
rustc --version
RUSTTORCH_BENCH_HARDWARE="your CPU model; your GPU model or CPU-only" \
  cargo bench -p rusttorch-data --bench data_loader --locked
```

Preserve the output with the machine's OS version and power configuration when
sharing results. Each row reports the dataset, batch/queue settings and named
baseline; the header records architecture, compiler and runtime versions.
This generated in-memory workload measures loader overhead, not image decoding
or training throughput. Automatic pinning is a no-op when CUDA is unavailable;
that result does not measure CUDA transfer performance. No timing threshold or
speedup claim is built into the benchmark.

## API and compatibility

Read the [Rust API reference](https://docs.rs/rusttorch-data) for complete item
contracts and examples. The [compatibility record](COMPATIBILITY.md) separately
tracks tested behavior against the project's pinned upstream reference. It is
an engineering audit, not a prerequisite for using RustTorch's data API.