Module source 

Data sources: typed sequences that drive workload iteration.

A source is a data provider with identity. It knows what it yields (schema), how much it has (extent), where the consumer is (cursor), and how to partition across concurrent fibers.

Sources replace the cycles counter as the workload iteration driver. The Polydat graph declares sources via the source keyword. The runtime pulls from sources to drive op dispatch. When a source is exhausted, the phase is done.

Source Types

• Range: range(0, N) — finite sequence of ordinals. Replaces cycles: N.
• Dataset: dataset_source("example:label_00", "base") — vectors, queries, etc.
• Derived: any Polydat binding promoted to a source via the source keyword.

Crate Sovereignty

All source API surface lives here in polydat. The runtime crates (host runtime, adapters) consume these types but don’t define them.

Structs

AndPolicy

Continue (extend) while ALL child policies say continue. Stops the moment any child policy returns None. The delta is the minimum of the children’s deltas — conservative step size keeps any single condition from over-shooting its stop point.

Cursors

Provenance-driven advancer that targets only the cursor nodes relevant to a specific set of output fields.

ExtendingRangeSourceFactory

Factory for ExtendingRangeSource. Differs from RangeSourceFactory in that end is an atomic that the extension policy may grow over the lifetime of the phase. global_extent() returns the CURRENT end so phase-status displays reflect any growth honestly.

ExtensionContext

Snapshot of cursor state passed to an ExtensionPolicy at each end-reach decision point. Policies are pure predicates over this context — they don’t carry their own clocks or counters.

OrPolicy

Continue (extend) while ANY child policy says continue. Stops only when every child policy returns None. The delta is the maximum of the policies that said continue — matches the most aggressive child still pushing forward.

OrdinalBatchLease

Ownership token for one range reserved from a perfect ordinal source.

RangeSourceFactory

Factory for range sources. Shared atomic cursor distributes ordinals across fibers. Replaces CycleSource.

SourceItem

A single item yielded by a source.

SourceReplayContract

Runtime source capability used by offset-stamped batch plans.

SourceSchema

Schema describing what a source yields.

TimeElapsedPolicy

Back-compat alias for the original time-only policy. Constructs an UntilElapsedPolicy with delta = base.

UntilCountPolicy

Extend by delta while ctx.consumed < min_count.

UntilElapsedPolicy

Extend while ctx.elapsed_ms < min_ms, projecting the remaining work from the observed rate and committing it in one batch.

UntilPassesPolicy

Extend by delta while ctx.passes() < min_passes. Passes are whole multiples of ctx.base.

Enums

CursorBatchError

Why a cursor batch could not be reserved.

CursorKind

Cursor-construction discriminator. Set by the Polydat compiler when it recognises a cursor’s constructor expression (range(...), until_elapsed(...), etc.); read by the executor at phase setup to instantiate the matching data-source factory + policy.

SourceReplayStability

Whether a source item can be reconstructed from its ordinal without consulting or advancing mutable source state.

SourceValueForm

A compact description of values which can be generated without rendering an opaque source item first.

Traits

DataSource

Consumption API for data sources. One instance per fiber.

DataSourceFactory

Factory that creates per-fiber DataSource readers.

ExtensionPolicy

Policy that decides whether to extend an ExtendingRangeSource when its current end is reached.