Struct Pipeline 

pub struct Pipeline {
    pub id: Uuid,
    pub name: Option<String>,
    pub description: String,
    pub tasks: Vec<TaskInfo>,
    pub retry_policy: RetryPolicy,
    pub batch_size: usize,
    pub data_id_fn: Option<Arc<dyn Fn(Arc<dyn Value>) -> Option<String> + Sync + Send>>,
    pub concurrency: usize,
    pub telemetry_settings: Option<Map<String, Value>>,
    pub rate_limiter: Option<Arc<dyn RateLimiter>>,
}

Fields

id: Uuid

name: Option<String>

Human-readable pipeline name (used as key for status tracking).

description: String

tasks: Vec<TaskInfo>

retry_policy: RetryPolicy

batch_size: usize

Default maximum number of items collected from an iterator / stream before dispatching them to the next task (individually for non-batch tasks, as a slice for batch tasks). Individual tasks can override this via TaskInfo::batch_size.

data_id_fn: Option<Arc<dyn Fn(Arc<dyn Value>) -> Option<String> + Sync + Send>>

Optional function to extract a stable data ID from input values. When set together with an [ExecStatusManager] on the context, the executor will skip items that are already completed.

concurrency: usize

Maximum number of data items processed concurrently through the full task chain. Default 1 = strictly sequential (current behaviour). Values > 1 use buffer_unordered for data-item-level parallelism.

telemetry_settings: Option<Map<String, Value>>

Optional pre-built telemetry settings snapshot (the | config merge from Python’s pipeline lifecycle events). When None, Pipeline Run * analytics events emit with no settings merged in. Populated by cognee-lib from Config::telemetry_snapshot().

Carried as a plain field rather than a feature-gated one so the Pipeline struct shape is stable across feature flips. The snapshot is only consumed when the telemetry feature is on.

rate_limiter: Option<Arc<dyn RateLimiter>>

Pipeline-wide proactive rate limiter applied to every task call (both single-value via call_with_retry and batch via dispatch_batch). Individual tasks may override it via TaskInfo::rate_limiter. None means no throttling.

Implementations

impl Pipeline

pub fn new(description: impl Into<String>) -> Pipeline

pub fn with_name(self, name: impl Into<String>) -> Pipeline

pub fn with_task(self, task: impl Into<TaskInfo>) -> Pipeline

pub fn with_retry(self, policy: RetryPolicy) -> Pipeline

pub fn with_batch_size(self, size: usize) -> Pipeline

pub fn with_data_id( self, f: Arc<dyn Fn(Arc<dyn Value>) -> Option<String> + Sync + Send>, ) -> Pipeline

Set the function used to extract a stable data ID from input values for incremental deduplication.

pub fn with_concurrency(self, n: usize) -> Pipeline

Set the number of data items processed concurrently. Default is 1 (sequential). When n > 1, items are processed in parallel via buffer_unordered(n).

Note: output order is not guaranteed when concurrency > 1.

pub fn with_rate_limiter(self, rl: Arc<dyn RateLimiter>) -> Pipeline

Set a pipeline-wide proactive rate limiter. Individual tasks may override it via TaskInfo::with_rate_limiter.

The limiter is acquired inside call_with_retry once per attempt (so each retry is a fresh acquisition) and once per batch call in dispatch_batch. Use this for LLM API quota throttling or per-host crawl-rate control.

See crate::rate_limiter for the distinction between this, Pipeline::with_concurrency (item parallelism), and RetryPolicy (reactive backoff).

pub fn with_telemetry_settings(self, settings: Map<String, Value>) -> Pipeline

Attach a pre-built telemetry settings snapshot (the | config merge for Pipeline Run Started/Completed/Errored analytics events). See Pipeline::telemetry_settings for details.