pub struct LoaderState<D, S, T = (), C = ()> {Show 13 fields
pub schema_version: u32,
pub dataset_identity: String,
pub epoch: u64,
pub iterator_generation: u64,
pub next_batch: u64,
pub next_logical_sample: u64,
pub dataset: D,
pub sampler: S,
pub transform: T,
pub collate: C,
pub rng_derivation_version: u32,
pub worker_seed_derivation_version: u32,
pub configuration: LoaderConfiguration,
}Expand description
Serializable input position and pipeline state for an exact map-loader restart.
Capture this on the iterator, after the application has successfully used a
batch. Recreate the same dataset and options, deserialize the state, and pass
it to crate::DataLoaderBuilder::resume_from. The next iterator resumes
before the first unconsumed batch. Prefetched samples are read again from
saved component state; sample payloads are not serialized.
§Save and restore a prefetched loader
The immutable rows below can safely be fetched again. A content version
identifies the data, while the loader validates sampling, batching, worker
and component configuration during restore. This example uses serde_json;
add it as a dependency when choosing JSON in your application.
use std::convert::Infallible;
use rusttorch_data::{DataLoader, Dataset, ReplaySafeDataset, ReplaySafeMap, VecCollate};
struct Rows([i64; 5]);
impl Dataset for Rows {
type Sample = i64;
type Error = Infallible;
fn len(&self) -> usize { self.0.len() }
fn get(&self, index: usize) -> Result<i64, Infallible> { Ok(self.0[index]) }
}
impl ReplaySafeDataset for Rows {}
fn main() -> Result<(), Box<dyn std::error::Error>> {
let build = || DataLoader::builder(ReplaySafeMap::new(Rows([2, 3, 5, 7, 11])))
.batch_size(2).workers(2).prefetch_factor(2).collate(VecCollate)
.dataset_identity("prime-rows-v1".to_owned());
let mut loader = build().build()?;
let mut iteration = loader.iter();
assert_eq!(iteration.next().unwrap()?, [2, 3]);
let saved = serde_json::to_string(&iteration.checkpoint()?)?;
let uninterrupted = iteration.collect::<Result<Vec<_>, _>>()?;
let mut restored = build().resume_from(serde_json::from_str(&saved)?).build()?;
let remaining = restored.iter().collect::<Result<Vec<_>, _>>()?;
assert_eq!(remaining, [vec![5, 7], vec![11]]);
assert_eq!(remaining, uninterrupted);
Ok(())
}Save model and optimizer state at the same training step; this state covers
only the input pipeline. Capture initially or after a successful batch, before
observing an error or exhaustion. Do not change the selected epoch before
consuming a pending resume: set_epoch intentionally starts a fresh pass.
Serial checkpoints support replay-safe or transactional datasets. Worker
checkpoints require replay-safe datasets and ordered delivery, with no byte
budget, timeout, custom worker initializer or persistent workers. Custom
transforms and collation need explicit checkpoint contracts. Use
crate::TensorDataset::into_replay_safe for isolated serial tensor data.
Fields§
§schema_version: u32State schema version.
dataset_identity: StringCaller-defined identity for the exact dataset contents.
epoch: u64Active sampler epoch.
iterator_generation: u64Transport generation. Serial state is always zero.
next_batch: u64Next consumer-visible batch number.
next_logical_sample: u64Number of logical sample occurrences already consumed.
dataset: DDataset component state.
sampler: SActive sampler or batch-source cursor state.
transform: TSerial transform state or a versioned worker transform envelope.
collate: CEffective coordinator collator/converter state.
rng_derivation_version: u32Task-RNG derivation version.
worker_seed_derivation_version: u32Worker-seed derivation version.
configuration: LoaderConfigurationExact loader configuration identity.
Trait Implementations§
Source§impl<D: Clone, S: Clone, T: Clone, C: Clone> Clone for LoaderState<D, S, T, C>
impl<D: Clone, S: Clone, T: Clone, C: Clone> Clone for LoaderState<D, S, T, C>
Source§fn clone(&self) -> LoaderState<D, S, T, C>
fn clone(&self) -> LoaderState<D, S, T, C>
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source. Read more