pub trait WorkerSourceFactory:
Send
+ Sync
+ 'static {
type Sample: Send + 'static;
type Error: Send + 'static;
type Source: Iterator<Item = Result<WorkerRecord<Self::Sample>, Self::Error>> + Send + 'static;
// Required method
fn create(&self, worker: WorkerContext) -> Result<Self::Source, Self::Error>;
// Provided method
fn exact_len(&self) -> Option<usize> { ... }
}Expand description
Open an independent input shard for each worker and each iteration.
Implement this for sharded files, partitioned database scans or other sources
that can be opened separately. Use context.info.id and
context.info.num_workers to select disjoint records; the loader does not
partition an arbitrary source automatically. Returning the same full stream
from every worker duplicates its records.
StreamDataLoaderBuilder provides a complete lazy modulo-shard example.
Retain WorkerContext when a decoder needs cancellation/deadline information.
Report the total record count across all shards through Self::exact_len
only when it is exact; return None for an unknown-length source.
Required Associated Types§
Required Methods§
Provided Methods§
Dyn Compatibility§
This trait is dyn compatible.
In older versions of Rust, dyn compatibility was called "object safety".