Trait Source 

pub trait Source: Send + Sync {
    // Required methods
    fn open(&self) -> Result<Box<dyn SourceIterator>>;
    fn name(&self) -> &str;

    // Provided method
    fn len_hint(&self) -> Option<u64> { ... }
}

A source of training data.

Sources are lazy - they describe WHERE data comes from, not the data itself. Data is only loaded when the pipeline pulls from the source.

Implementing a Source

use tenshift_core::source::{Source, SourceIterator};
use tenshift_core::sample::Sample;
use tenshift_core::error::Result;

struct MyDatabase {
    connection_string: String,
}

impl Source for MyDatabase {
    fn open(&self) -> Result<Box<dyn SourceIterator>> {
        // Open connection, return iterator over rows
    }

    fn len_hint(&self) -> Option<u64> {
        None // unknown size
    }

    fn name(&self) -> &str {
        "my_database"
    }
}

Required Methods

fn open(&self) -> Result<Box<dyn SourceIterator>>

Create an iterator over samples from this source.

Called once per epoch. The iterator is consumed by the pipeline’s worker threads.

Errors

Returns an error if the source cannot be opened (e.g., file not found, network unreachable, invalid credentials).

fn name(&self) -> &str

Human-readable name for this source (for logging and error messages).

Provided Methods

fn len_hint(&self) -> Option<u64>

Optional hint about the total number of samples.

Used for progress reporting and pre-allocation. Return None if unknown.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl<T> Source for Box<T>

where T: Source + ?Sized,

fn open(&self) -> Result<Box<dyn SourceIterator>>

fn len_hint(&self) -> Option<u64>

fn name(&self) -> &str

Implementors

impl Source for GlobSource

impl Source for JsonlSource

impl Source for MemorySource

impl<A: AsyncSource + 'static> Source for AsyncToSyncAdapter<A>

impl<S> Source for DistributedSampler<S>

where S: Source,