Skip to main content

Module executor

Module executor 

Source
Expand description

Run a list of ExpandedNodes under a bounded-concurrency executor.

Semantics:

  • Roots run concurrently under Semaphore(max_concurrent).
  • Each root captures its written records (via a CapturingSink wrapper) so descendants can fan out per parent record.
  • For each child whose parent has finished successfully, one pipeline invocation runs per parent record. ${parent.dotted.path} tokens in the source / sink config and state-key suffix are resolved against that record via interpolate_record.
  • All invocations share one global semaphore — children and roots compete for the same budget.
  • A node with depends_on: [row, …] starts only after every listed row’s invocations finish successfully — pure completion-ordering, no record hand-off. A failed or skipped dependency skips the node (and, in turn, its own subtree and dependents).
  • on_error: continue (default) skips a failed node’s subtree but keeps running siblings. on_error: stop cancels everything after the first failure.
  • State-key collisions among children of the same parent surface as a CliError::DuplicateStateKey.

Structs§

ExecuteOptions
Knobs passed to run_expanded.
InvocationOutcome
One pipeline invocation’s outcome.
RunSummary
Aggregate outcome of run_expanded.

Functions§

build_dlq_config
Translate a crate::config::DlqSpec from the YAML/JSON config into a runtime DlqConfig ready to attach to a Pipeline.
run_expanded
Execute every node in nodes. nodes must be in BFS order (roots first then children) — that’s what crate::expand::expand returns.