Skip to main content

StreamLoaderState

Struct StreamLoaderState 

Source
pub struct StreamLoaderState<S, T = (), C = ()> {
Show 15 fields pub schema_version: u32, pub source_identity: String, pub factory_kind: String, pub epoch: u64, pub source_generation: u64, pub transport_generation: u64, pub next_batch: u64, pub next_sequence: u64, pub short_tail: bool, pub source_length: Option<usize>, pub rng_derivation_version: u32, pub worker_seed_derivation_version: u32, pub configuration: StreamCheckpointConfiguration, pub lanes: Vec<StreamLaneState<S, T>>, pub coordinator: C,
}
Expand description

Serializable input position and component state for resuming a sharded stream.

Obtain this from crate::ExactStreamLoaderIter::checkpoint and pass the decoded value to crate::StreamDataLoaderBuilder::resume. Store it with the model, optimizer and training counters from the same completed batch. It contains shard cursors and transform/collator state, not decoded records or model tensors. crate::ExactStreamDataLoader demonstrates JSON round-trip and exact replay; applications choose their storage format and atomic writes.

Fields§

§schema_version: u32

Stream schema version; currently one.

§source_identity: String

Caller identity for exact source contents.

§factory_kind: String

Stable factory kind/version.

§epoch: u64

Task epoch.

§source_generation: u64

Logical source and transform initialization generation.

§transport_generation: u64

Transport generation, advanced by each checkpoint barrier.

§next_batch: u64

Next visible batch index.

§next_sequence: u64

Next expected global sequence ID.

§short_tail: bool

Whether the last visible batch was the final incomplete global tail.

§source_length: Option<usize>

Exact global source length when supplied by the factory.

§rng_derivation_version: u32

Task RNG derivation version.

§worker_seed_derivation_version: u32

Worker seed derivation version.

§configuration: StreamCheckpointConfiguration

Exact runtime settings.

§lanes: Vec<StreamLaneState<S, T>>

Canonically ordered worker states; never contains decoded samples.

§coordinator: C

Coordinator collation state.

Trait Implementations§

Source§

impl<S: Clone, T: Clone, C: Clone> Clone for StreamLoaderState<S, T, C>

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl<S: Debug, T: Debug, C: Debug> Debug for StreamLoaderState<S, T, C>

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl<'de, S, T, C> Deserialize<'de> for StreamLoaderState<S, T, C>
where S: Deserialize<'de>, T: Deserialize<'de>, C: Deserialize<'de>,

Source§

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>
where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer. Read more
Source§

impl<S, T, C> Serialize for StreamLoaderState<S, T, C>
where S: Serialize, T: Serialize, C: Serialize,

Source§

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>
where __S: Serializer,

Serialize this value into the given Serde serializer. Read more

Auto Trait Implementations§

§

impl<S, T, C> Forget for StreamLoaderState<S, T, C>

§

impl<S, T, C> Freeze for StreamLoaderState<S, T, C>
where Vec<StreamLaneState<S, T>>: Freeze, C: Freeze,

§

impl<S, T, C> RefUnwindSafe for StreamLoaderState<S, T, C>

§

impl<S, T, C> Send for StreamLoaderState<S, T, C>
where Vec<StreamLaneState<S, T>>: Send, C: Send,

§

impl<S, T, C> Sync for StreamLoaderState<S, T, C>
where Vec<StreamLaneState<S, T>>: Sync, C: Sync,

§

impl<S, T, C> Unpin for StreamLoaderState<S, T, C>
where Vec<StreamLaneState<S, T>>: Unpin, C: Unpin,

§

impl<S, T, C> UnsafeUnpin for StreamLoaderState<S, T, C>

§

impl<S, T, C> UnwindSafe for StreamLoaderState<S, T, C>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> DeserializeOwned for T
where T: for<'de> Deserialize<'de>,

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V