Skip to main content

CheckpointableSource

Trait CheckpointableSource 

Source
pub trait CheckpointableSource: Iterator<Item = Result<WorkerRecord<Self::Sample>, Self::Error>> {
    type Sample;
    type State: Clone + Serialize + DeserializeOwned;
    type Error;

    // Required methods
    fn error_sequence(&self, error: &Self::Error) -> SequenceId;
    fn snapshot(&self) -> Self::State;
    fn validate_snapshot(&self, state: &Self::State) -> Result<(), Self::Error>;
    fn restore_validated(&mut self, state: &Self::State);

    // Provided method
    fn set_run_context(&mut self, _context: WorkerContext) { ... }
}
Expand description

A stream reader whose next unread position can be saved and restored exactly.

Implement this for seekable files, versioned record stores, or other sources where replaying a read produces the same record. The state must contain every cursor and decoder setting that affects later output; keep it owned and independent of mutable buffers. A destructive queue cannot promise this contract merely by remembering how many items were read.

crate::ExactStreamDataLoader demonstrates a lazy source, cursor validation, JSON state, and a restart that yields the same remaining batches.

Validation is read-only; applying accepted state must not fail or panic. Records must have strictly increasing per-shard global sequence IDs, and all shards together must cover the ordered global sequence without gaps. If the source retains a WorkerContext, implement Self::set_run_context so a checkpoint barrier can replace the cancelled context before further reads.

Required Associated Types§

Source

type Sample

Owned decoded sample.

Source

type State: Clone + Serialize + DeserializeOwned

Owned state, independent of future decoder progress.

Source

type Error

Preserved source error.

Required Methods§

Source

fn error_sequence(&self, error: &Self::Error) -> SequenceId

Returns the failed read’s stable global position without changing state.

Called after next() returns an error. This position must obey the shard’s strictly increasing sequence contract and repeat after rollback.

Source

fn snapshot(&self) -> Self::State

Captures the boundary before the next read attempt.

Source

fn validate_snapshot(&self, state: &Self::State) -> Result<(), Self::Error>

Validates without changing the source.

Source

fn restore_validated(&mut self, state: &Self::State)

Applies state previously accepted by validation.

Provided Methods§

Source

fn set_run_context(&mut self, _context: WorkerContext)

Replaces the run cancellation/deadline context before resumed reads.

Sources retaining a context must override this method and replace it. Logical worker generation and seed remain unchanged. This must not fail.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§