Skip to main content

WorkerSourceFactory

Trait WorkerSourceFactory 

Source
pub trait WorkerSourceFactory:
    Send
    + Sync
    + 'static {
    type Sample: Send + 'static;
    type Error: Send + 'static;
    type Source: Iterator<Item = Result<WorkerRecord<Self::Sample>, Self::Error>> + Send + 'static;

    // Required method
    fn create(&self, worker: WorkerContext) -> Result<Self::Source, Self::Error>;

    // Provided method
    fn exact_len(&self) -> Option<usize> { ... }
}
Expand description

Open an independent input shard for each worker and each iteration.

Implement this for sharded files, partitioned database scans or other sources that can be opened separately. Use context.info.id and context.info.num_workers to select disjoint records; the loader does not partition an arbitrary source automatically. Returning the same full stream from every worker duplicates its records.

StreamDataLoaderBuilder provides a complete lazy modulo-shard example. Retain WorkerContext when a decoder needs cancellation/deadline information. Report the total record count across all shards through Self::exact_len only when it is exact; return None for an unknown-length source.

Required Associated Types§

Source

type Sample: Send + 'static

Sample produced by every shard.

Source

type Error: Send + 'static

Typed source creation or iteration failure.

Source

type Source: Iterator<Item = Result<WorkerRecord<Self::Sample>, Self::Error>> + Send + 'static

Independently owned shard iterator.

Required Methods§

Source

fn create(&self, worker: WorkerContext) -> Result<Self::Source, Self::Error>

Creates this worker’s shard for the supplied generation context.

Provided Methods§

Source

fn exact_len(&self) -> Option<usize>

Returns the exact global record count when known.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§