swale

A scheduled, dependency-ordered orchestrator of asset graphs.

swale runs on the taquba durable execution crates. Every task instance runs as a single workflow run, and the record of its outcome and the event for the scheduler commit in one transaction.

Definition file

One TOML file per graph. An asset node declares the asset it produces and the assets it consumes, and a task node declares its upstream nodes with after.

[graph]
name = "orders_daily"
schedule = "0 2 * * *"
partition = "daily"

[[node]]
name = "extract"
produces = "orders_raw"
operator = "shell"
[node.params]
command = "dbt run --select orders_raw >&2 && printf '{\"table\": \"orders_raw\"}'"

[[node]]
name = "load"
produces = "orders_warehouse"
consumes = ["orders_raw"]
operator = "http"
[node.params]
method = "POST"
url = "https://warehouse.example/load"
body = '{"table": "{{ upstream.extract.table }}"}'

The definition module documents every field, and the loader reports every fault of a definition at once. From Rust, load_path returns the checked graph:

use std::path::Path;
use swale::OperatorSet;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let graph = swale::load_path(
        Path::new("examples/orders_daily.toml"),
        &OperatorSet::builtin(),
    )?;
    println!("{}: {} nodes", graph.name(), graph.nodes().len());
    Ok(())
}

Operators

An operator runs a node, and its parameters are the params table of the node. A parameter string refers to {{ partition }}, {{ run.<field> }} (id or summary), {{ upstream.<node>.<path> }} and {{ env.<NAME> }}. The upstream reference is a path into the JSON output of an upstream node, and the env reference is an environment variable of the daemon.

subprocess runs argv with a JSON document on stdin (the identity, the parameters and the upstream outputs) and reads the output from stdout.
shell runs command with sh -c, with the identity in SWALE_* variables, the upstream outputs in SWALE_INPUTS and the env table in the environment, and reads the output from stdout.
http sends method, url, headers and body within timeout and outputs {"status": <code>, "body": <value>}, with a JSON body parsed.
object_exists waits until an object exists at url, a store URL, with a poll every interval and a failure after timeout. No worker is held between polls, and the output is the URL, the size and the last-modified time of the object.

Exit code 0 and a 2xx status are success. Exit code 75, a 5xx status, a 429 status, a connection failure and a timeout are transient errors, retried up to retries times. Any other exit code or status is a permanent error, which dead-letters the task instance. A program or a request must be idempotent per attempt.

Schedule

The daemon fires the schedule of a graph, a cron expression with five fields in UTC. The partition of a firing is the start of the schedule interval that ends at the firing time: a daily schedule at 02:00 that fires on 2026-09-16 runs the partition 20260915. A graph with a schedule declares partition as daily or hourly.

The first adoption of a graph runs the partitions of the firings within its catchup window. After downtime the daemon replays the missed firings within the same window.

Command

The swale binary is enabled by the default cli feature. A program that embeds the library can set default-features = false.

swale validate loads a definition and prints each fault, one per line.

swale run runs a graph for one partition on a store, prints each node's record as it is written and waits for the run to settle. A second run for the same partition resumes the existing graph run, and a process interrupted mid-run resumes without repeating a completed node.

swale publish writes a definition to the store as the current definition of its graph. swale daemon runs the published graphs until it is interrupted. It adopts each published definition within one sync interval, fires each schedule and runs the task instances. A graph run keeps the definition it started from, so an edit applies from the next graph run. The default pool always exists, and --pool name=steps adds a pool.

The daemon keeps the records of a settled graph run and the request records for --retention, ninety days by default. A start of a partition whose records were removed runs the graph again.

Every command that opens a store takes --store: a directory or an object store URL (s3://bucket/prefix, gs://bucket/prefix, az://container/prefix). A cloud scheme needs the matching cargo feature (aws, gcp or azure) and reads the provider's environment variables for its credentials. The default store is ~/.swale/store.

swale run and swale daemon open the store as its only writer. A swale run on the store of a running daemon opens a second writer, and the store then refuses the writes of the daemon.

swale status lists the graphs of a store with the count of their graph runs in each state. swale status <graph> lists the latest graph runs of the graph, and swale status <graph> <partition> lists the nodes of one graph run. A node without a record is ready, waiting or blocked. A blocked node does not run until a rerun changes the record of an upstream.

swale queues lists the job counts of every queue, and swale queues <queue> lists the dead jobs of one queue. Both commands only read from the store, so they can run alongside a daemon.

swale start <graph> <partition>... starts the graph run of each partition, swale rerun <graph> <partition> <node> runs a node again, and swale cancel <graph> <partition> cancels an active graph run. Each command writes a request to the store and does not open the queue, and the daemon applies the request at its next sync pass. With --wait the command waits for the outcome and prints it.

A rerun of a failed or cancelled node runs the node again, and its downstream nodes follow as their trigger rules allow. A rerun of a succeeded node runs the node again and, after it, every node downstream of it through an asset edge or an all-succeeded edge, each with the new outputs. A task node with the all-done or one-failed rule keeps its record. Until such a node runs again, the status view shows it as ready, waiting or blocked with its last record.

$ swale validate examples/orders_daily.toml
orders_daily: 5 nodes, 3 edges
$ swale run examples/local.toml --store ./swale-store
local/none: started, 1 root node(s) submitted
  first: succeeded (local-none-first-r0)
  second: succeeded (local-none-second-r0)
local/none: complete
$ swale status local none --store ./swale-store
local/none: complete, requested 2026-09-17T13:48:19Z, definition 9450f82715de
NODE    POOL     STATE      RUN                   TERMINATED
first   default  succeeded  local-none-first-r0   2026-09-17T13:48:19Z
second  default  succeeded  local-none-second-r0  2026-09-17T13:48:20Z
$ swale publish examples/orders_daily.toml --store s3://bucket/swale
orders_daily: published 36e831ff15e026ad45115374cb98edec6b617dffb0d4ef40f9fd351ea36bca9a
$ swale daemon --store s3://bucket/swale --pool warehouse=2
$ swale rerun orders_daily 20260915 transform --store s3://bucket/swale --wait
request 01K5ARRE8ZW9K8XTJTQ6PVYJK7: submitted orders_daily-20260915-transform-r1

A command exits with status 0, 1 or 2:

0. The command succeeded. An interrupt ends swale daemon with this status.
1. A definition has a fault, a graph run failed, a request was refused or another error occurred.
2. The arguments are not valid.

License