Skip to main content

DataSourceFactory

polydat_core::iteration::source

Trait DataSourceFactory 

Source 
pub trait DataSourceFactory: Send + Sync {
    // Required methods
    fn create_reader(&self) -> Box<dyn DataSource>;
    fn schema(&self) -> &SourceSchema;
    fn global_consumed(&self) -> u64;

    // Provided methods
    fn global_extent(&self) -> Option<u64> { ... }
    fn replay_contract(&self) -> SourceReplayContract { ... }
    fn rewind_for_poll(&self) -> bool { ... }
}
Expand description

Factory that creates per-fiber DataSource readers.

Holds shared state (atomic cursor, partition pool) that’s distributed across readers. Each fiber gets its own reader.

§Dispatch model

The stride is the stanza length — the number of source items a fiber acquires as an atomic unit. One stanza of ops processes one stride of source items. Strides are inseparable: a fiber that acquires a stride processes all items before acquiring the next.

The default implementation (RangeSourceFactory) uses a shared atomic cursor — all fibers pull strides from the same counter, producing natural monotonic striping. This is correct for range sources where items are independent ordinals.

For dataset sources with locality benefits (mmap prefetch), factories can implement partitioned allocation: each fiber gets a pre-assigned range of strides, and when exhausted, steals strides from a shared pool. The stride is the minimum unit of work stealing — a fiber never steals partial stanzas.

Required Methods§

Source

fn create_reader(&self) -> Box<dyn DataSource>

Create a new reader for a fiber.

Source

fn schema(&self) -> &SourceSchema

Schema for all readers from this factory.

Source

fn global_consumed(&self) -> u64

Global items consumed across all readers (for progress reporting).

Provided Methods§

Source

fn global_extent(&self) -> Option<u64>

Known extent, if finite. Same as schema().extent but avoids clone.

Source

fn replay_contract(&self) -> SourceReplayContract

Replay/addressability capability shared by readers from this factory.

Source

fn rewind_for_poll(&self) -> bool

Rewind the factory’s shared cursor to the start so a subsequent create_reader() produces a fresh stream covering the same ordinal range: what a host that re-runs a source between poll rounds calls after each round exhausts it.

Default impl: returns false to signal the factory doesn’t support rewinding. Factories that DO support it (RangeSourceFactory, ExtendingRangeSourceFactory) override and reset their internal cursor / extent state. A host that needs to rewind should reject a factory that returns false with a clear diagnostic rather than complete silently after the first round.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§