Struct PipelineBuilder 

pub struct PipelineBuilder<I, O>
where
    I: Value,
    O: Value,
{ /* private fields */ }

A compile-time type-safe builder for Pipeline.

PipelineBuilder<I, O> tracks the input type of the first task (I) and the output type of the most recently added task (O) as type parameters. The add_task method accepts only a TypedTask<O, O2>, ensuring that each task’s input type matches the previous task’s output type. When all tasks have been added, call build to erase the types and obtain a Pipeline that the standard executor can run.

Example

let pipeline = PipelineBuilder::new_with_task(
        "my pipeline",
        TypedTask::sync(|s: &String, _| Ok(Box::new(s.len()))),
    )
    .add_task(TypedTask::sync(|n: &usize, _| Ok(Box::new(format!("len={n}")))))
    .with_name("length-formatter")
    .build();

Implementations

impl<I, O> PipelineBuilder<I, O>

where I: Value, O: Value,

pub fn new_with_task( description: impl Into<String>, first_task: TypedTask<I, O>, ) -> PipelineBuilder<I, O>

Create a new builder, seeding it with the first task.

The type parameters I and O are inferred from first_task.

pub fn add_task<O2>(self, task: TypedTask<O, O2>) -> PipelineBuilder<I, O2>

where O2: Value,

Append a task whose input type must equal the current output type O.

Returns a new builder whose output type is updated to O2. The compile-time constraint TypedTask<O, O2> ensures type safety: passing a task with a mismatched input type is a compile error.

pub fn add_task_named<O2>( self, task: TypedTask<O, O2>, name: impl Into<String>, ) -> PipelineBuilder<I, O2>

where O2: Value,

Append a task with an explicit human-readable name.

Equivalent to add_task followed by setting the resulting TaskInfo::name. The name is what the executor’s stamp_tree_dyn writes into DataPoint.source_task and what the PipelineWatcher lifecycle hooks observe.

pub fn with_first_task_name( self, name: impl Into<String>, ) -> PipelineBuilder<I, O>

Set the human-readable name of the first task already pushed by new_with_task.

Used by builders that want to name the seed task without restructuring the constructor. The name is what the executor’s stamp_tree_dyn writes into DataPoint.source_task.

Panics

Panics if no tasks have been pushed yet (impossible via the public API, since new_with_task always seeds one).

pub fn with_name(self, name: impl Into<String>) -> PipelineBuilder<I, O>

Set a human-readable name (used as key for status tracking).

pub fn with_retry(self, policy: RetryPolicy) -> PipelineBuilder<I, O>

Set the retry policy applied to all tasks.

pub fn with_batch_size(self, size: usize) -> PipelineBuilder<I, O>

Set the default batch size (items accumulated before dispatching to a batch task).

pub fn with_concurrency(self, n: usize) -> PipelineBuilder<I, O>

Set the number of data items processed concurrently.

pub fn with_data_id( self, f: Arc<dyn Fn(Arc<dyn Value>) -> Option<String> + Sync + Send>, ) -> PipelineBuilder<I, O>

Set the function used to extract a stable data ID for incremental deduplication.

pub fn build(self) -> Pipeline

Consume the builder and produce a Pipeline.

Type safety is fully enforced by the time build is called; the returned Pipeline uses the existing type-erased executor unchanged.