pub struct StreamLoaderState<S, T = (), C = ()> {Show 15 fields
pub schema_version: u32,
pub source_identity: String,
pub factory_kind: String,
pub epoch: u64,
pub source_generation: u64,
pub transport_generation: u64,
pub next_batch: u64,
pub next_sequence: u64,
pub short_tail: bool,
pub source_length: Option<usize>,
pub rng_derivation_version: u32,
pub worker_seed_derivation_version: u32,
pub configuration: StreamCheckpointConfiguration,
pub lanes: Vec<StreamLaneState<S, T>>,
pub coordinator: C,
}Expand description
Serializable input position and component state for resuming a sharded stream.
Obtain this from crate::ExactStreamLoaderIter::checkpoint and pass the
decoded value to crate::StreamDataLoaderBuilder::resume. Store it with the
model, optimizer and training counters from the same completed batch. It
contains shard cursors and transform/collator state, not decoded records or
model tensors. crate::ExactStreamDataLoader demonstrates JSON round-trip
and exact replay; applications choose their storage format and atomic writes.
Fields§
§schema_version: u32Stream schema version; currently one.
source_identity: StringCaller identity for exact source contents.
factory_kind: StringStable factory kind/version.
epoch: u64Task epoch.
source_generation: u64Logical source and transform initialization generation.
transport_generation: u64Transport generation, advanced by each checkpoint barrier.
next_batch: u64Next visible batch index.
next_sequence: u64Next expected global sequence ID.
short_tail: boolWhether the last visible batch was the final incomplete global tail.
source_length: Option<usize>Exact global source length when supplied by the factory.
rng_derivation_version: u32Task RNG derivation version.
worker_seed_derivation_version: u32Worker seed derivation version.
configuration: StreamCheckpointConfigurationExact runtime settings.
lanes: Vec<StreamLaneState<S, T>>Canonically ordered worker states; never contains decoded samples.
coordinator: CCoordinator collation state.