Module source 

Source traits for synchronous and asynchronous data origins. Source trait - the extension point for data origins.

A [Source] produces [Sample]s from some origin (filesystem, network, database, etc.). Community contributors add new sources by implementing this single trait.

Built-in sources include GlobSource, MemorySource, and DistributedSampler, with optional feature-gated sources such as CSV and io_uring readers.

Async Sources

For async data origins, implement [AsyncSource] and wrap with Pipeline::from_async_source. This runs the async source in a dedicated Tokio runtime without blocking the pipeline’s worker threads.

Structs

AsyncIteratorAdapter

Runs AsyncSourceIterator asynchronously block by block.

AsyncToSyncAdapter

Adapts an AsyncSource into a synchronous Source by running a Tokio runtime.

Traits

AsyncSource

Async-capable source of training data.

AsyncSourceIterator

Async iterator over samples from a source.

Source

A source of training data.

SourceIterator

Iterator over samples from a source.

Type Aliases

BoxFuture

Boxed future used by async source extension traits.