durable-rust

Lightweight durable job execution engine backed by Postgres.

No external services, no replay log, no orchestrator. Just checkpointed steps in plain Rust.

Quick example

use durable::{Ctx, DurableError};

async fn ingest(db: &DatabaseConnection) -> Result<(), DurableError> {
    let ctx = Ctx::start(db, "ingest", None).await?;

    // Step 1 — runs once, result saved to Postgres
    let shards: Vec<u32> = ctx.step("resolve_shards", || async {
        Ok(vec![0, 1, 2, 3])
    }).await?;

    // Step 2..N — skips already-completed shards on resume
    for shard in shards {
        ctx.step(&format!("shard_{shard}"), || async move {
            process_shard(shard).await
        }).await?;
    }

    // Child workflow with its own steps
    let child = ctx.child("post_process", None).await?;
    child.step("notify", || async { send_slack("done").await }).await?;
    child.complete(&"ok").await?;

    ctx.complete(&"finished").await?;
    Ok(())
}

If the process crashes at shard 2, on restart it skips shards 0-1 (results saved) and resumes from 2.

Schema

Two tables in a durable schema:

• task -- unified row for workflows, steps, and child workflows. Self-referential via parent_id. Idempotent creation via UNIQUE(parent_id, name).
• task_queue -- optional concurrency and rate-limit controls.

See doc/schema.md for the full DDL, indexes, and status lifecycle.

Getting started

1. Start Postgres:

   docker compose -f compose.db.yml up -d
   

2. Add the dependency (path or git):

   [dependencies]
   durable = { path = "crates/durable" }
   

3. Initialize in your app -- connects and runs migrations automatically:

   let db = durable::init("postgres://durable:durable@localhost:5432/durable").await?;
   

4. Write workflows using Ctx:

   • Ctx::start(&db, name, input) -- create or resume a root workflow
   • ctx.step(name, closure) -- run-once step with saved output
   • ctx.child(name, input) -- spawn a nested child workflow
   • ctx.complete(output) -- mark workflow done

Crate structure

Run the example

docker compose -f compose.db.yml up -d
cargo run -p nested-etl

The nested-etl example runs a parent ETL workflow that spawns child workflows per data source, then demonstrates crash recovery by resuming mid-run.

Run tests

cargo test --workspace

Design docs

• doc/api.md -- API design with proc macros (#[durable::workflow], #[durable::step])
• doc/dataflow.md -- dataflow diagrams for direct, queued, scheduled, and nested execution
• doc/schema.md -- full schema DDL and status lifecycle

Design principles

1. Postgres is the source of truth -- no WAL, no event log, no separate replay mechanism
2. Steps are idempotent by design -- if a step completed, its saved result is returned; the closure is never re-executed
3. No orchestrator -- the job runner is just your application code calling ctx.step()
4. No serialization framework -- uses serde_json for input/output
5. Crash safe -- incomplete steps are detected on resume; completed steps replay from saved output
6. Observable -- query the durable.task table directly for monitoring and debugging