Struct PipelineConfig 

pub struct PipelineConfig {

    pub version: u32,
    pub name: Option<String>,
    pub vars: Option<HashMap<String, Value>>,
    pub params: ParamsSpec,
    pub auth: Option<HashMap<String, Value>>,
    pub pipeline: PipelineSpec,
    pub matrix: Vec<MatrixRow>,
    pub execution: Option<ExecutionSpec>,
    pub selection: Option<SelectionSpec>,
    pub observability: Option<ObservabilitySpec>,
    pub delivery: DeliveryMode,
    pub resilience: Option<ResilienceSpec>,
    pub sla: Option<SlaSpec>,
    pub reconcile: Option<ReconcileSpec>,
    pub shard: Option<ShardingSpec>,
    pub replication: Option<ReplicationSpec>,
    pub backfill: Option<BackfillSpec>,
    pub partition: Option<PartitionSpec>,
    pub schedule: Option<ScheduleSpec>,
    pub lineage: Option<LineageConfig>,
    pub catalog: Option<CatalogSpec>,
    pub notifications: Vec<NotificationSpec>,
    pub metadata_columns: Option<MetadataColumnsSpec>,
}

Top-level pipeline definition.

Fields

version: u32

Config-format version. Currently always 1.

name: Option<String>

Optional human-readable name (used in logs and error messages).

vars: Option<HashMap<String, Value>>

Optional shared constants. Resolvable as ${vars.key} anywhere in the config (including inside named templates). Resolved at load time, after env/file/secret substitution.

params: ParamsSpec

Optional typed run parameters (#444) — the config’s trigger-time override surface. Each entry declares a name, scalar type, required/default, a secret flag, and a description; values are referenced as ${param.NAME} and bound before parsing by crate::params::bind (from --param, faucet template run, or POST /v1/templates/{id}/runs). Because binding happens pre-parse, no ${param.*} token ever reaches a connector. See faucet schema params.

auth: Option<HashMap<String, Value>>

Optional named auth providers. Each entry is a { type, config } spec (the same shape as inline auth) built once and shared across every connector that references it via auth: { ref: <name> }. Values are kept as raw JSON so faucet-auth owns the per-type schema.

pipeline: PipelineSpec

Base pipeline — every matrix row is deep-merged into this.

matrix: Vec<MatrixRow>

Matrix of per-row overrides. Empty or omitted means “one anonymous row” (full pipeline runs once with no merge).

execution: Option<ExecutionSpec>

Optional execution controls (concurrency, on-error policy).

selection: Option<SelectionSpec>

Optional matrix-row selection policy (#377). Currently carries only include_parents — how a selected row’s parent: / depends_on: ancestors are resolved when they are not independently in the run set. Absent = the built-in default (include_parents: off, strict).

observability: Option<ObservabilitySpec>

Optional observability configuration (Prometheus + tracing).

delivery: DeliveryMode

Delivery guarantee for every row (overridable per matrix row). Default at_least_once — no behaviour change for existing configs. exactly_once requires an idempotent sink, a deterministic-replay source, and a state store (enforced at expand time).

resilience: Option<ResilienceSpec>

Optional resilience policy (retry / backoff / circuit-breaker / poison-pill). Top-level in v1 (not per-matrix-row). Absent = no behaviour change. Consumed by faucet run/schedule/replicate.

sla: Option<SlaSpec>

Optional data-freshness & volume SLA (#202). Top-level in v1 (not per-matrix-row, like resilience:). Evaluated after every root invocation by faucet run/schedule/serve/replicate; violations emit metrics + warnings and never fail the run. Staleness/volume checks require a state: block (enforced at expand time).

reconcile: Option<ReconcileSpec>

Optional completeness reconciliation (#502): after a successful root run, compare rows written against an authoritative count probe and fail the run on a shortfall beyond tolerance — a silent-truncation guard, especially for write_mode: overwrite.

shard: Option<ShardingSpec>

Optional source-shard distribution for clustered (Mode B) execution. Only consumed by faucet serve --cluster: a run whose source is shardable is split into shard.count shards processed concurrently across cluster workers. Ignored by faucet run and by a non-cluster serve, so it is fully backward compatible.

replication: Option<ReplicationSpec>

Optional snapshot→CDC replication block. Consumed only by faucet replicate; ignored by faucet run (like schedule:).

backfill: Option<BackfillSpec>

Optional backfill defaults (window/concurrency/timezone, #282). Consumed only by faucet backfill; ignored by faucet run (like schedule: / replication:). See faucet schema backfill.

partition: Option<PartitionSpec>

Default range partitioning applied to every root row that does not declare its own (#479). Unlike backfill: / schedule:, this IS consumed by faucet run — a partitioned row fans out on every run.

schedule: Option<ScheduleSpec>

Available on crate feature schedule only.

Optional cron schedule. Only consumed by faucet schedule; ignored by faucet run. Presence makes the config runnable on a schedule.

lineage: Option<LineageConfig>

Available on crate feature lineage only.

Optional OpenLineage emission. Consumed by faucet run/schedule/serve.

catalog: Option<CatalogSpec>

Available on crate feature catalog only.

Optional Data Movement Catalog store (#279): where faucet run / schedule / replicate accumulate the cross-run dataset catalog (identity, schema timeline, volume/freshness, lineage edges). faucet serve ignores this block and records into its --history backend. Recording never fails a run. See faucet schema catalog.

notifications: Vec<NotificationSpec>

Available on crate feature notify only.

Optional notification / incident-routing rules (#280). Fan pipeline lifecycle and health events (run failure/success, SLA breach, circuit open, contract abort, DLQ threshold, scheduler stuck) out to Slack / PagerDuty / a signed webhook. Consumed by every runtime (run/schedule/serve/replicate); delivery never fails a run.

metadata_columns: Option<MetadataColumnsSpec>

Optional _faucet_* run/lineage metadata columns (#510) stamped onto every row by a connector-agnostic sink decorator (works for any sink). Off unless present.

Implementations

impl PipelineConfig

pub fn from_path( path: impl AsRef<Path>, profile: Option<&str>, ) -> CliResult<Self>

Load a pipeline config from disk. The file extension determines the parser: .yaml / .yml → YAML, .json → JSON. Other extensions are rejected.

Composition runs first via crate::compose::compose: extends (base inheritance), profiles (the named overlay selected by profile), and !include (YAML fragment substitution) are resolved into a single merged document before ${...} interpolation and parsing.

Secret directives (${vault:…}, ${aws-sm:…}, etc.) are not resolved by this path. If any are present the call returns CliError::SecretsRequireAsyncLoad — use Self::from_path_async instead.

pub fn from_path_with( path: impl AsRef<Path>, profile: Option<&str>, inputs: &RunInputs, ) -> CliResult<Self>

Self::from_path with caller-supplied RunInputs (params: values and an ${env:} overlay, #444).

pub fn from_path_tolerating_secrets( path: impl AsRef<Path>, profile: Option<&str>, ) -> CliResult<Self>

Like Self::from_path but does not reject secret directives — they are left unresolved. Used by validate --no-secrets. Composition (extends/profiles/!include) runs first via crate::compose::compose.

pub fn from_path_tolerating_secrets_with( path: impl AsRef<Path>, profile: Option<&str>, inputs: &RunInputs, ) -> CliResult<Self>

Self::from_path_tolerating_secrets with caller-supplied RunInputs.

pub async fn from_path_async( path: impl AsRef<Path>, profile: Option<&str>, ) -> CliResult<Self>

Async load path: like Self::from_path but resolves secret-manager directives (${vault:…}, ${aws-sm:…}, …) as a final stage. Composition (extends/profiles/!include) runs first via crate::compose::compose.

pub async fn from_path_async_with( path: impl AsRef<Path>, profile: Option<&str>, inputs: &RunInputs, ) -> CliResult<Self>

Self::from_path_async with caller-supplied RunInputs (params: values and an ${env:} overlay, #444).

pub fn from_text(text: &str, path: &Path) -> CliResult<Self>

Parse an already-interpolated config string. path is only used for error messages and to pick the parser by file extension.

pub fn from_value(value: Value) -> CliResult<Self>

Build a config from an already-parsed JSON value (used by faucet serve, which merges a submitted body onto a --default-config base). Runs the same version check + structural ${...} ref resolution as Self::from_text.

Note: load-time ${env:VAR} / ${file:PATH} / ${secret:VAR} directives are not resolved here — the caller must pre-resolve them (e.g. by running interpolate on the source text) before building the Value.

Trait Implementations

impl Clone for PipelineConfig

fn clone(&self) -> PipelineConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PipelineConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for PipelineConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl JsonSchema for PipelineConfig

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl Serialize for PipelineConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.