Runledger

Runledger is a durable job queue and workflow engine for Rust, backed by PostgreSQL.

You bring concrete job handlers and a Postgres database; Runledger gives you a persistent queue, a worker runtime with leasing and retries, cron schedules, and a first-class workflow DAG for multi-step work with dependencies, fan-out/fan-in, and external (human or API) approval gates. State lives entirely in your database, so there is no broker to run and nothing to lose on restart.

The crates are libraries: you embed them in your own service and supply the handlers, process model, and admin surface.

Features

• Durable Postgres-backed queue — enqueue, claim, heartbeat, retry with provider-directed timing, succeed, cancel, dead-letter, and requeue jobs. Survives restarts; no separate broker.
• Worker runtime — a Supervisor that runs worker, scheduler, and reaper loops with lease-based ownership, lease expiry recovery, and graceful shutdown.
• Workflow DAGs — model dependent work declaratively. The engine validates the graph, enqueues root steps, releases dependents as prerequisites finish, and keeps run status coherent across cancellation and external gates.
• Bounded continuation and replay — continue successful work in resumable slices, recover canceled or dead-lettered direct jobs, replay successful direct jobs without mutating their history, and recover terminal workflows as new lineage-linked runs.
• Durable coordination — reusable active-workflow keys and lease-fenced single-permit execution resources coordinate work across workers and organizations.
• Workflow results — designate a result step, persist compact JSON output, and read or wait for it through a scoped workflow handle.
• External gates — pause a workflow on a human approval or third-party callback and resume it with complete_external_workflow_step.
• Cron schedules — recurring, UTC, idempotently materialized entrypoints.
• Idempotent enqueue — keyed jobs and workflow runs deduplicate against the original enqueue request.
• Catalog-driven setup — register handlers, sync job definitions, and declare schedules from one source of truth at startup.
• Operator TUI — a read-only terminal dashboard for queue metrics, jobs, workflows, and definitions.
• Offline builds — SQLx compile-time-checked queries with a committed .sqlx/ cache, so the workspace builds without a live database.

Contents

• Workspace crates
• Installation
• Quick start
• Core concepts
  • Choosing the right API
  • Workflow DAGs
  • Workflow results and handles
  • Handler-selected retry timing
  • Bounded job continuation
  • Active workflow keys
  • Durable execution resources
  • Workflow recovery
  • Upgrade map for releases 0.6 through 0.9
  • 0.7 to 0.8 activation and rollback
  • Schedules
  • Job definition catalog
• Examples
• Admin reads
• Operator TUI
• Configuration
• Database schema and migrations
• Operational notes
• PostgreSQL requirements
• Working in this repository
• Releasing
• Repository layout
• License

Workspace crates

Crate	Role
runledger-core	Storage-agnostic contracts: handler traits, runtime types, statuses, identifiers, and workflow enqueue/DAG validation. No persistence or async loops.
runledger-postgres	SQLx-backed PostgreSQL persistence: queue and job lifecycle, schedules, the workflow DAG state machine, runtime configs, logs, and admin reads/mutations.
runledger-runtime	The async runtime: Supervisor, worker/scheduler/reaper loops, the job catalog, the handler registry, and runtime configuration.
runledger-tui	Read-only terminal UI for monitoring queue metrics, jobs, workflows, and definitions.
runledger-test-support	Published test utilities for ephemeral PostgreSQL databases and scoped environment overrides.

runledger-core, runledger-postgres, and runledger-runtime are the libraries you depend on. Keep the layering intact: contracts in core, runtime orchestration in runtime, and SQL/state-machine logic in postgres.

Installation

Add the libraries to your service:

[dependencies]
runledger-core = "0.9"
runledger-postgres = "0.9"
runledger-runtime = "0.9"

[dev-dependencies]
runledger-test-support = "0.9"

The published crates require Rust 1.88+ and PostgreSQL 18+. Older PostgreSQL releases are not supported, even when an extension supplies an equivalent uuidv7() function. See PostgreSQL requirements.

Common imports:

use runledger_core::prelude::*;
use runledger_postgres::prelude::*;
use runledger_runtime::prelude::*;

Quick start

Downstream services typically run a web/API process that enqueues work and a separate worker process that runs handlers against the same database. A minimal worker:

use std::time::Duration;

use runledger_core::jobs::{JobCompletion, JobContext, JobFailure, JobType};
use runledger_core::prelude::async_trait;
use runledger_runtime::Supervisor;
use runledger_runtime::catalog::JobCatalog;
use runledger_runtime::config::JobsConfig;
use runledger_runtime::registry::JobHandler;
use serde_json::Value;
use sqlx::postgres::PgPoolOptions;

struct SendEmail;

#[async_trait]
impl JobHandler for SendEmail {
    fn job_type(&self) -> JobType<'static> {
        JobType::new("jobs.email.send")
    }

    async fn execute(&self, _context: JobContext, _payload: Value) -> Result<JobCompletion, JobFailure> {
        // do the work
        Ok(JobCompletion::success())
    }
}

async fn run_worker() -> Result<(), Box<dyn std::error::Error>> {
    let pool = PgPoolOptions::new()
        .connect(&std::env::var("DATABASE_URL")?)
        .await?;

    // Apply the bundled schema (or validate it; see "Database schema and migrations").
    runledger_postgres::migrate_after_idempotency_cutover(&pool).await?;

    // Register handlers and sync their job definitions.
    let catalog = JobCatalog::new().job("jobs.email.send", SendEmail);
    catalog.sync_definitions(&pool).await?;

    // Run the supervisor until Ctrl-C, with a 30s shutdown drain deadline.
    let supervisor = Supervisor::builder(&pool, JobsConfig::from_env())?
        .with_catalog(&catalog)
        .build()?;
    let shutdown_result = supervisor
        .run_until_shutdown(
            async {
                if let Err(error) = tokio::signal::ctrl_c().await {
                    eprintln!("failed to listen for shutdown signal: {error}");
                }
            },
            Duration::from_secs(30),
        )
        .await;

    // Keep pool cleanup independent from the shutdown result.
    pool.close().await;
    shutdown_result?;
    Ok(())
}

From anywhere else (such as your API), enqueue a job against the same pool:

let job = runledger_postgres::jobs::enqueue_job(&pool, /* JobEnqueue */).await?;

Notes on the worker lifecycle:

• run_until_shutdown() is the preferred facade for worker binaries: it observes internal task failures while still applying a shutdown deadline. When the deadline is hit, remaining supervised tasks are aborted and in-flight handler futures are dropped.
• Treat any error from run_until_shutdown(), shutdown(), or shutdown_with_timeout() as fatal for the process — a supervised loop panicked, exited before shutdown was requested, or did not observe shutdown within the deadline.
• Size the shutdown timeout to cover handler drain time under JobsConfig::max_global_concurrency and your database capacity. A per-handler high-percentile latency is a reasonable starting point.
• Capture the shutdown result before closing the pool, so cleanup runs even when shutdown reports an error.
• worker::run_worker_loop, scheduler::run_scheduler_loop, and reaper::run_reaper_loop remain available as low-level building blocks for custom orchestration; they return RuntimeLoopExit (JoinHandle<RuntimeLoopExit> if you type join handles explicitly).

A typical host application:

1. Either call migrate_after_idempotency_cutover(&pool) to apply the bundled schema, or apply migrations with your own tooling and then call ensure_schema_compatible_after_idempotency_cutover(&pool) to validate it.
2. Create a shared sqlx::PgPool.
3. Register handlers in a JobCatalog (or directly in a JobRegistry for advanced setups).
4. Run the Supervisor in a worker process.
5. Call runledger_postgres::jobs::* from your own admin/API surfaces.

This workspace deliberately stops at the library boundary; it does not prescribe your process model or handler packaging. A compile-checked worker skeleton lives at runledger-runtime/examples/worker_binary.rs.

Core concepts

A job is one independent, retried unit of work, identified by a job type and carrying a JSON payload. A workflow run is a DAG of steps (each step is a job) with dependency edges; the engine drives it to completion. A schedule is a cron entrypoint that materializes jobs over time. An external step is a workflow step that blocks until something outside the system completes it.

Choosing the right API

Use the highest-level API that matches the shape of the work. This matters especially for agents and generated integrations: a workflow DAG is a built-in feature, not something to recreate by polling jobs or chaining handlers by hand.

Need	Prefer
One independent retried unit of work	runledger_postgres::jobs::enqueue_job
Multi-step work with dependencies	WorkflowDagBuilder (simple DAGs), or WorkflowRunEnqueueBuilder / WorkflowStepEnqueueBuilder (advanced), then enqueue_workflow_run
Multi-step work with a durable JSON result	Declare a result step, enqueue with enqueue_workflow_run_handle, then call WorkflowRunHandle::get_result
Fan-out, fan-in, or ordered stages	WorkflowDagBuilder::after_success / after_terminal (or lower-level depends_on_success / depends_on_terminal)
Human/API approval or another external gate	External workflow steps and complete_external_workflow_step
Delayed or recurring entrypoint	JobScheduleUpsert and upsert_job_schedule (or catalog schedules)
Provider-directed retry lower bound	JobFailure::retry_not_before_delay or retry_not_before
More work after one successful bounded slice	JobCompletion::continue_now or continue_after; workflow steps must opt in
At most one active workflow for an application key	WorkflowRunEnqueueBuilder::active_key and enqueue_or_get_active_workflow
Mutual exclusion for jobs sharing one external resource	enqueue_job_with_execution_resource or WorkflowStepEnqueueBuilder::execution_resource
Recover a canceled or dead-lettered direct job	compare_and_requeue_job with exact observed state
Intentionally repeat a successful direct job	compare_and_replay_succeeded_job
Recover a terminal workflow without rewriting history	recover_workflow_run
Worker process lifecycle	runledger_runtime::Supervisor::run_until_shutdown
Admin/status views	runledger_postgres::jobs read/list/count APIs, including count_workflow_runs

For ordinary dependent work, do not poll get_job_by_id in a loop, enqueue dependent jobs from parent handlers, encode dependency state in payload JSON, or add app-owned tables to track workflow edges. Model the run as a workflow DAG instead. Hand-rolled orchestration is only appropriate when you are intentionally building an orchestrator outside Runledger.

For prompt-facing summaries, see llms.txt (short) and docs/downstream-agent-guide.md (longer).

Workflow DAGs

Model dependencies directly in the enqueue request. The engine persists the run, validates the DAG, enqueues root steps, releases dependents as prerequisites finish, and keeps run status coherent with cancellation and external gates.

use runledger_core::jobs::WorkflowDagBuilder;

let metadata = serde_json::json!({"source": "api"});
let crawl_payload = serde_json::json!({"profile_id": "p_123"});
let classify_payload = serde_json::json!({"profile_id": "p_123"});
let score_payload = serde_json::json!({"profile_id": "p_123"});
let persist_payload = serde_json::json!({"profile_id": "p_123"});

let run = WorkflowDagBuilder::new("profiles.research", &metadata)
    .idempotency_key("profile:p_123:research")
    .job("crawl", "profiles.crawl", &crawl_payload)?
    .job("classify", "profiles.classify", &classify_payload)?
    .after_success("classify", ["crawl"])?
    .job("score", "profiles.score", &score_payload)?
    .after_success("score", ["crawl"])?
    .job("persist", "profiles.persist", &persist_payload)?
    .after_success("persist", ["classify", "score"])?
    .build()?;

let workflow_run = runledger_postgres::jobs::enqueue_workflow_run(&pool, &run).await?;

WorkflowDagBuilder takes raw string identifiers for readable call sites and validates the workflow shape before enqueueing — but it does not prove at compile time that a job type has a registered definition or handler. Reach for WorkflowRunEnqueueBuilder / WorkflowStepEnqueueBuilder when you need per-step priority, attempts, timeout, or stage; external steps; hand-authored dependency specs; or explicit StepKey / JobType values.

Validation happens in two stages — some errors surface at the call site, the rest at .build() / .try_build():

Call	Fails immediately	Deferred until build
WorkflowDagBuilder::new(...)	never	blank workflow type
WorkflowDagBuilder::try_new(...)	blank workflow type	empty step list, dependency graph errors
.job(step, job_type, payload)	blank step key, blank job type, duplicate step key	job-type registration is not checked here
.after_success(step, prereqs) / .after_terminal(...)	blank target/prerequisite key, unknown target step	missing prerequisite, self-dependency, duplicate dependency, cycle
.idempotency_key(...)	never	blank idempotency key

Workflow results and handles

Workflows can declare one DAG step as the durable result step. A handler returns a compact JSON result with JobCompletion::with_output(...); when the run reaches SUCCEEDED, Runledger materializes that step output as the workflow result.

let run = WorkflowDagBuilder::new("profiles.research", &metadata)
    .idempotency_key("profile:p_123:research")
    .job("crawl", "profiles.crawl", &crawl_payload)?
    .job("persist", "profiles.persist", &persist_payload)?
    .after_success("persist", ["crawl"])?
    .result_step("persist")?
    .build()?;

let handle = runledger_postgres::jobs::enqueue_workflow_run_handle(&pool, &run).await?;
let result = handle.get_result(Default::default()).await?;

The handle is scoped when created or retrieved: organization workflows use WorkflowRunHandleScope::Organization, global workflows use Global, and trusted operator surfaces can use Admin. Use get_status for a cheap status probe, get_run to load the scoped run record, and get_result to wait for or read the declared result. Notifications wake waiters quickly, but polling remains the correctness path. WorkflowRunWaitOptions::default() waits up to five minutes by default; set timeout: None only when the caller intentionally wants to wait indefinitely. Each active waiter may hold a PostgreSQL LISTEN connection until the result is ready, so size pools accordingly and use shorter explicit timeouts for high fan-out callers. Keep outputs compact: result JSON is persisted on the job, step, and workflow run rows; store large artifacts externally and return references. Workflows without a declared result still run normally; get_result returns workflow.result_not_declared. Other handle error codes include workflow.handle_storage_error, workflow.run_not_found, workflow.result_missing, workflow.result_unsuccessful_terminal, and workflow.result_wait_timeout.

External workflow steps can also provide result output when completed successfully:

use runledger_core::jobs::{StepKey, WorkflowStepStatus};
use runledger_postgres::jobs::CompleteExternalWorkflowStepInput;

let approval_output = serde_json::json!({ "approved_by": "ops" });

runledger_postgres::jobs::complete_external_workflow_step(
    &pool,
    &CompleteExternalWorkflowStepInput {
        workflow_run_id,
        organization_id: None,
        step_key: StepKey::new("approval"),
        terminal_status: WorkflowStepStatus::Succeeded,
        status_reason: Some("approved"),
        last_error_code: None,
        last_error_message: None,
        output: Some(&approval_output),
    },
)
.await?;

output is valid only with WorkflowStepStatus::Succeeded; failed or canceled external completions must pass None. Retrying completion for an already terminal external step is idempotent only when the terminal status, status_reason, last_error_code, and last_error_message match; changed metadata returns workflow.external_step_conflicting_completion_retry. For successful completions, output must also match, or Runledger returns workflow.external_step_conflicting_output_retry.

Breaking API note: JobHandler::execute returns Result<JobCompletion, JobFailure>. The old stage-bearing JobProgress completion type was removed; use JobCompletion::success() or JobCompletion::with_output(...). In-flight progress reporting still uses JobProgressUpdate. Completion disposition and final output are intentionally private; inspect them with disposition() / output() and use constructors rather than struct literals.

Handler-selected retry timing

When a provider supplies a dynamic reset time, a handler can attach either a relative delay or an absolute UTC timestamp to a retryable failure:

use std::time::Duration;

let transport_failure = JobFailure::retryable(
    "provider.temporarily_unavailable",
    "Provider asked the client to retry later.",
)
.retry_not_before_delay(Duration::from_secs(30));

let rate_limit_failure = JobFailure::retryable(
    "provider.rate_limited",
    "Provider rate limit reached.",
)
.retry_not_before(provider_reset_at);

In 0.8, JobFailure gained private retry-timing state and can no longer be constructed with a struct literal. Use JobFailure::new, retryable, terminal, timeout, lease_expired, or panicked, then add a lower bound with the methods above. The older retry_after and retry_at names are deprecated because the requested time never overrides a later policy backoff. Low-level persistence integrations must construct the now non-exhaustive JobFailureUpdate with JobFailureUpdate::new(...) and optionally .with_retry_timing(...).

This is a failed attempt, not an attempt-neutral defer or a successful continuation. It consumes the current run's attempt budget, keeps the same run_number, and does not release workflow dependents. If the failure remains retryable, Runledger computes ordinary policy backoff from the registered job-type/failure-code override or exponential fallback, then schedules the later of that policy time and the handler's lower bound. Terminal and panicked failures, and failures that exhaust max_attempts, are dead-lettered without applying or validating timing.

retry_not_before_delay is measured from the PostgreSQL completion clock; positive sub-millisecond values round up to one millisecond. Zero and absolute times before PostgreSQL's range supply no additional lower bound. A winning hint that cannot be represented becomes the terminal job.invalid_retry_timing handler failure. retry_not_before uses the supplied provider timestamp, rounded up only when needed for PostgreSQL microsecond precision. A past hint cannot shorten the ordinary policy delay. Future timestamps outside PostgreSQL's supported range become the terminal job.invalid_retry_timing handler failure.

Retry attempts retain the policy delay in retry_delay_ms and record requested_retry_not_before, effective_next_run_at, and retry_timing_source (POLICY or HANDLER_NOT_BEFORE). The same audit fields are written to the RETRY_SCHEDULED event, which retains requested_retry_at and next_run_at as legacy aliases. Observer dispositions report the committed effective schedule, while JobFailure::retry_timing() remains the handler's request.

Bounded job continuation

A handler that has successfully finished one bounded slice but still has more work for the same logical job can return JobCompletion::continue_now() or JobCompletion::continue_after(delay). Direct jobs may return this disposition immediately. Workflow job steps require an explicit, persisted enqueue-time opt-in:

let step = WorkflowStepEnqueueBuilder::new(step_key, job_type, &payload)
    .allow_handler_continuation()
    .try_build()?;

The workflow-step default is false, and external steps cannot opt in. This keeps rollout scoped and prevents an accidental handler continuation from creating an indefinitely active workflow. Handlers that keep returning terminal success or failure retain their existing behavior. Progress and checkpoints can be carried into the next run with the existing builders:

use std::time::Duration;

let completion = JobCompletion::continue_after(Duration::from_secs(5))
    .progress(processed, total)
    .checkpoint(serde_json::json!({ "cursor": next_cursor }));

On the next claim, the handler reads that committed value from context.checkpoint; the original payload remains unchanged. A first run, or a run without committed resume state, receives None.

Runledger closes the current attempt successfully, changes the exact live lease back to PENDING, increments run_number, resets attempt to zero, releases the worker/lease, and writes a REQUEUED event whose reason and stable requeue_kind are HANDLER_CONTINUATION. The job ID and payload stay the same. A later claim of the next run starts at attempt one with a fresh failure-attempt budget. If the continuation write does not commit, the durable row remains leased and normal lease recovery retries the idempotent slice while attempts remain or dead-letters an exhausted run. State from an uncommitted slice, including its new checkpoint, cannot be recovered. Final output is valid only for terminal success; continuation-plus-output cannot be constructed or deserialized. For an opted-in workflow step, the same transaction returns the step from RUNNING to ENQUEUED. The workflow run remains active and dependencies stay blocked; only a later terminal success, failure, or cancellation releases dependency edges and recomputes terminal workflow state. A workflow step without the persisted opt-in still converts an accidental continuation into the terminal job.workflow_handler_continuation_not_enabled handler failure.

A mixed 0.7/0.8 worker fleet is unsafe after continuation-enabled workflow steps are emitted: a 0.7 worker can claim one and terminalize it when the handler returns continuation. Deploy all workers, reapers, schedulers, and administrative processes, then wait for old processes and leases to quiesce before enabling a canary workflow job type.

Successful continuations deliberately have no implicit run cap and do not consume the per-run max_attempts failure budget. The handler owns its terminal condition; use a nonzero delay for polling-style work and do not return continue_now() forever. Production handlers should version their checkpoint shape, make every slice idempotent, enforce a logical deadline or run limit, canary activation by job type or tenant, and alert on continuation rate and run depth. get_job_continuation_metrics returns a JobContinuationMetricsRecord per job type with continued_24h, active_continued_count, and max_active_run_number for canary and runaway-loop alerts. Active counts include only jobs whose current run was created by a handler continuation; a later admin recovery is not mislabeled as active continuation. The packaged external-consumer smoke test is a compile-checked continuation, recovery, successful-replay, and metrics example; the downstream agent guide contains the full adoption checklist and PostgreSQL 18 operational queries.

Lifecycle observers receive on_job_continued(JobContinuedEvent) after the same run's on_job_running callback settles, with the completed run identity, duration, next run number/time, and committed progress. A failed continuation write is reported through JobCompletionPersistFailedEvent with JobCompletionPersistenceOperation::Continuation. Observers are best-effort and different run numbers may be observed concurrently or out of order; correlate by (job_id, run_number) and use durable job events for authoritative history.

Active workflow keys

Use WorkflowRunEnqueueBuilder::active_key(...) with enqueue_or_get_active_workflow when only one active cycle may exist in a global or organization scope. Always match the explicit EnqueueActiveWorkflowOutcome: Inserted, ExistingActive, or ExistingIdempotent. Scope does not include workflow type: namespace active keys by workflow type unless different workflow types should deliberately coordinate, because ExistingActive may otherwise return the other type's run. An active claim is durable and is not reusable until the prior workflow is terminal and any canceled live lease has quiesced. Active keys must be non-blank and at most 512 bytes. Deferred cancellation release is performed by the lease reaper; if the reaper is disabled or stopped, the key remains reserved until reaping resumes. ExistingActive can therefore carry a terminal canceled run; treat the outcome, rather than terminal status alone, as the reuse decision. enqueue_workflow_run_handle intentionally rejects active-key payloads because a handle alone would discard that classification. After matching the active enqueue outcome, create a handle with workflow_run_handle using the returned run ID and matching scope.

Durable execution resources

For one-permit concurrency across otherwise unrelated jobs, enqueue a direct job with enqueue_job_with_execution_resource or configure a workflow job step with .execution_resource("provider-account:123"). Resource reservation occurs atomically before lease creation. Blocked jobs stay PENDING at attempt zero and do not consume a returned worker claim slot. Ownership is fenced to the exact run, attempt, worker, and lease, and releases on success, failure, continuation, prestart claim release, reaping, or quiesced cancellation. Resource keys must contain a non-whitespace character and are limited to 512 bytes. The direct-job API returns JobEnqueueOutcome; when the job also has an idempotency_key, its execution resource is part of the canonical enqueue request, so retrying that request with a different resource is a conflict. Execution resource keys are global across organizations: namespace keys in the application when tenants must not contend, and reuse a key across organizations only when they intentionally share one external capacity limit. Keys guarantee mutual exclusion, not global FIFO: a type-restricted worker selects the oldest eligible job within its allowed types. Mixing filtered and unfiltered workers, or workers with different type filters, can therefore reorder contenders for a shared key without weakening mutual exclusion. A requested batch size is an upper bound, not a fullness guarantee: concurrent workers that race for the same keys can return short batches even while unrelated work exists. Each poll examines a bounded resource-head window (1,024–16,384 eligible keyed jobs, scaled by requested batch size), so an unusually dense same-key prefix can also return a short batch instead of scanning an unbounded backlog. Resource inserts use consistent key order to reduce cross-filter deadlock risk. Exclusivity is lease-scoped: the reaper releases an expired owner when it transitions the owning job, so provider-side operations must still be fenced or idempotent in case a handler outlives its lease after heartbeat loss. Successful direct-job replay preserves the source execution resource. If the reaper is disabled or stopped, expired claims remain reserved until it resumes; cleanup is bounded by the configured reaper batch limit. Lease transitions commit before coordination-claim cleanup, so a cleanup failure cannot roll back successfully reaped jobs. ReapExpiredLeasesDetailedResult reports released active and resource claim counts plus typed cleanup errors; the runtime logs failures and warns when either cleanup reaches its batch limit. Heartbeating a resource owner also renews its durable claim, adding one keyed-row update per heartbeat. Continuation releases the claim between slices, so the next slice re-contends for the resource instead of retaining it across runs.

Workflow recovery

recover_workflow_run never reopens terminal steps. It creates a distinct run and a workflow_recoveries lineage row, reconstructing the DAG from the source's canonical enqueue snapshot plus committed append history. Reusing the same (source_run_id, request_key) with identical fields returns the existing recovery; conflicting reuse fails. Recovery reacquires the source active key and preserves per-step execution resources, handler-continuation opt-ins, and the source's resolved priority, attempt limit, and timeout even if job-definition defaults have changed. It uses each source step's latest persisted payload, so an operator correction made through the pending-step payload API is not replaced by the older canonical enqueue payload. The new run does not reuse the source's permanent workflow idempotency key; the recovery request key is its separate replay identity. Runs created before canonical snapshots were available, snapshots with unknown fields, and unsupported mutation kinds are rejected instead of being replayed ambiguously. Recovery request keys must be non-blank and at most 512 bytes. Retention cannot delete only a recovery run while its source remains, because doing so would erase the request's idempotency guard; a source-led statement may delete the complete lineage together. Every new workflow stores its canonical enqueue snapshot, including step payloads, so budget for that additional JSON storage and keep large artifacts behind references.

Construct the non-exhaustive request through its constructor. Omitting .organization_id(...) means an exactly global source; source_step_id is optional audit context and does not limit the full replay:

let request = WorkflowRecoveryRequest::new(
    source_run_id,
    recovery_request_key,
    WorkflowRecoveryMode::FullReplay,
    "operator-approved retry",
)
.organization_id(organization_id)
.source_step_id(failed_step_id);

let outcome = recover_workflow_run(&pool, &request).await?;
match outcome.disposition {
    WorkflowRecoveryDisposition::Inserted => { /* new recovery run */ }
    WorkflowRecoveryDisposition::Existing => { /* idempotent request retry */ }
    _ => { /* future disposition */ }
}

Use recover_workflow_run_tx only when recovery must compose with other application writes; the caller-owned transaction must be READ COMMITTED and the function neither commits nor rolls back it. A source must be terminal. Recovery of an active-key workflow can remain blocked until its old claim is quiescent, or while another run owns that key.

Upgrade map for releases 0.6 through 0.9

The 0.9 release line includes the contracts introduced in the preceding three releases. When skipping versions, preserve each release's schema and runtime fence:

Release	Schema requirement	Activation requirement
0.6	No new migration.	Deploy every job-state writer with continuation and typed recovery unused; wait for every pre-0.6 process and live lease to quiesce before activating either path. Migrate deprecated requeue_job callers to exact typed outcomes.
0.7	Apply 202607190001_job_replays_and_continuation_metrics before replay or metrics callers.	Successful replay and continuation metrics are additive. Use Runledger's filtered migration/schema helpers for expand-first deployment and code rollback.
0.8	Apply 202607250001_harden_continuation_metrics_payload_validation and 202607280001 through 202607280005 before any 0.8 runtime loop or persistence API runs.	Deploy every 0.8 writer with new paths unused, quiesce all older processes and leases, then canary workflow continuation, active keys, resources, retry hints, and workflow recovery.
0.9	No migration after 0.8.0.	Custom runtimes may adopt JobLeaseIdentity and its _for_lease lifecycle APIs without a coordinated schema or source migration; the positional functions remain available.

For 0.8 source upgrades, construct WorkflowDagStepValidationInput with WorkflowDagStepValidationInput::new(...) and its option setters; it is now non-exhaustive and includes handler-continuation and execution-resource settings. WorkflowStepDbRecord exposes the matching persisted fields, so direct struct-literal consumers must update. Construct the non-exhaustive workflow recovery request through WorkflowRecoveryRequest::new(...) as shown above, and keep wildcard arms when matching non-exhaustive recovery outcomes.

The next section summarizes the current transition. The downstream activation and rollback runbook contains the PostgreSQL 18 gates and the earlier direct-job recovery migration details.

0.7 to 0.8 activation and rollback

The 0.8 features use additive migrations and require a two-phase rollout:

1. Apply 202607250001_harden_continuation_metrics_payload_validation and the migrations through 202607280005_workflow_recoveries. Deploy 0.8 with workflow continuation emission, active-key enqueue, execution resources, retry hints, and recovery calls disabled to every process that can participate in job lifecycle state: workers, reapers, and admin, API, CLI, or repair processes that cancel or requeue jobs. Keep those paths unused until every 0.7 process has stopped and old leases have quiesced. If an upgrade skips releases, this means every pre-0.8 process.
2. Enable opted-in workflow continuation and the new enqueue/recovery paths by canary job type or tenant.

After resource-constrained jobs are emitted, a PostgreSQL trigger rejects any lease that lacks the exact durable resource claim. A 0.7 worker therefore fails loudly instead of silently violating mutual exclusion, but it can repeatedly roll back claim batches that encounter constrained work. The activation fence above remains mandatory for availability as well as protocol compatibility.

Before rollback, disable new 0.8 writes, drain continuation-created and resource-constrained work, wait for retained cancellation leases to quiesce, then stop all 0.8 writers before starting 0.7 processes. Starting a 0.7 worker while any resource-constrained job remains PENDING or LEASED causes its lease transaction to fail at the database fence.

Leave the additive 0.8 schema applied when the rollback binary uses Runledger's filtered startup helpers. A raw 0.7 MIGRATOR.run(...) rejects the newer SQLx history and requires the destructive down-migration path documented in the full runbook.

The target of .after_success(...) / .after_terminal(...) must already have been added with .job(...); prerequisite steps may be added later in the chain, as long as every referenced step exists before .build() succeeds.

Schedules

Schedules are UTC-only. Choose an API by who owns the schedule definition:

• .schedule(...) + sync_schedules — static schedules registered in the worker catalog next to their handler.
• sync_schedules_with — schedule specs assembled at startup from config, feature flags, or tenants (outside the builder chain).
• sync_schedules_exact / sync_schedules_exact_with — when this deployment owns a bounded schedule-name scope and missing schedules in that scope should be deactivated. Exact sync takes a bounded table lock so overlapping startup syncs do not interleave their active sets. Scheduler claims and fire-cursor updates can briefly wait behind the same lock; during rolling deploys, keep scopes narrow enough that old and new workers do not deactivate each other's schedules unintentionally. Keep owned scopes deployment-stable: feature-flagged schedules should usually stay registered with is_active: false instead of disappearing from the scope.
• job_schedule + upsert_job_schedule — one-off setup, migrations, admin tools, or schedules that should not be catalog-owned. Call set_job_schedule_active separately to change active state on an existing lower-level schedule.

use runledger_runtime::catalog::{CatalogJobScheduleSpec, JobCatalog};

let catalog = JobCatalog::new()
    .job("profiles.refresh", RefreshHandler)
    .schedule(CatalogJobScheduleSpec {
        name: "profiles.refresh.hourly",
        job_type: "profiles.refresh",
        cron_expr: "0 0 * * * *",
        payload_template: &serde_json::json!({}),
        is_active: flags.hourly_refresh,
        organization_id: None,
        max_jitter_seconds: 0,
        next_fire_at: None,
    });

catalog.sync_definitions(&pool).await?;
catalog.sync_schedules(&pool).await?;

Register a schedule's .job(...) before its .schedule(...) — schedule registration validates the referenced catalog job type immediately. Sync preserves an existing next_fire_at cursor while the cron expression is unchanged; changing cron_expr stores the spec's next_fire_at, or Utc::now() when it is None. Catalog schedule sync applies each spec's is_active value on every sync, so an admin pause made with set_job_schedule_active(false) is overwritten when the catalog spec still says is_active: true. Use the lower-level job_schedule + upsert_job_schedule path for schedules whose active state should be owned by admin pause/resume workflows; that path sets is_active on first insert, then preserves the stored active state on conflict.

Active schedules require enabled job definitions. Creating, syncing, or activating a schedule for a missing or disabled definition returns job_schedule.definition_not_found_or_disabled; disabling a job definition that still has active schedules returns job_definition.active_schedule_exists. During scheduler catch-up after downtime, Runledger materializes at most one stale fire with its original scheduled_for metadata, then coalesces next_fire_at to the first future cron fire instead of replaying every missed tick.

For exact sync of registered schedules, derive the owned scope from the catalog to avoid repeating names:

let scope = catalog.schedule_sync_scope()?;
catalog.sync_schedules_exact(&pool, &scope).await?;

If a deployment needs both registered schedules and dynamic startup specs in one exact source-of-truth set, build one explicit spec list and JobCatalogScheduleSyncScope for sync_schedules_exact_with; Runledger does not provide an implicit union helper because that can hide ownership mistakes.

Job definition catalog

sync_definitions is additive: it owns the definition fields it writes (version, retry limits, timeout, priority), restoring them to effective catalog values on each startup. It preserves an existing disabled row, so operator pauses survive restarts; an explicit enabled(false) default or per-job override disables a definition. Removed catalog entries are not deleted or disabled.

Use sync_definitions_exact with a JobCatalogSyncScope when startup should also disable enabled job_definitions rows that are absent from the catalog but inside an explicit owned job-type set. Exact sync returns the disabled job types, refuses to disable definitions still referenced by active schedules, and (unlike additive sync) restores catalog entries' enabled state from catalog defaults.

Override individual definitions with job_with_definition_overrides / definition_overrides:

let catalog = JobCatalog::new()
    .job_with_definition_overrides(
        "documents.extract",
        ExtractDocuments,
        JobCatalogDefinitionOverrides::new()
            .timeout_seconds(600)
            .priority(20),
    )
    .job_with_definition_overrides(
        "auth.cleanup",
        CleanupAuth,
        JobCatalogDefinitionOverrides::new()
            .timeout_seconds(60)
            .priority(0),
    );

Overrides take precedence over JobCatalogDefaults for only the fields they set: version, max_attempts, timeout_seconds, priority, and enabled. Version, attempts, and timeout values must be positive; priority may be zero or negative. An enabled(true) override can keep one job effectively enabled under disabled catalog defaults, while enabled(false) disables that job during sync. Additive sync still preserves an already-disabled database row for effectively enabled jobs so operator pauses survive restarts; exact sync restores enabled state from the effective catalog value.

Catalog helper builders validate catalog membership and effective enabled state; operator-disabled database rows are enforced later by persistence APIs (job enqueue, schedule materialization, workflow enqueue). The lower-level JobEnqueue, JobScheduleUpsert, WorkflowDagBuilder, and WorkflowStepEnqueueBuilder APIs remain available when you do not use a catalog.

Examples

These examples and integration references are compile-checked:

• Enqueue one job
• Workflow DAG (fan-out / fan-in)
• External workflow gate
• Append workflow steps
• Scheduled job entrypoint
• Worker binary skeleton
• Packaged continuation, retry timing, direct recovery, replay, and metrics smoke test
• Active-workflow key integration reference
• Execution-resource integration reference
• Workflow-recovery integration reference

Admin reads

The runledger_postgres::jobs admin surface exposes job/workflow detail, list, and count helpers for operator UIs and service-owned dashboards. Use list_workflow_runs with WorkflowRunListFilter when rendering workflow tables, and count_workflow_runs with WorkflowRunCountFilter for status counters such as failed workflows or runs waiting for external completion. These helpers use the same optional organization scope and workflow-type substring filtering as the TUI.

Use get_job_continuation_metrics for continuation canaries and runaway-loop alerts. Each JobContinuationMetricsRecord reports the prior 24 hours' successful continuations, the number of pending/leased jobs whose current run was created by continuation, and the highest current run number among those active jobs. Passing no organization filter aggregates all scopes; it does not mean exact global scope.

Durable event consumers should call list_job_events and prefer JobEventRecord::decoded_payload() for Runledger-authored continuation, administrative requeue, and successful-replay payloads. The decoded enums are non-exhaustive: keep wildcard arms and retain JobEventRecord::payload as the raw fallback for historical, malformed, custom, or future shapes instead of hand-parsing requeue_kind or replay lineage fields.

update_job_payload_uuid_array_field is intentionally narrow: it mutates one UUID-array payload field only for direct jobs that are still pending and unclaimed. It returns JobPayloadUuidArrayFieldUpdate::Updated, NotFound, or Rejected with a reason. Rejections distinguish workflow-managed jobs, idempotent request snapshots that cannot be kept consistent, and jobs that are already claimed or terminal.

Operator TUI

runledger-tui is a read-only terminal UI for operators and local development. It connects to the same database as your workers and surfaces dashboard metrics, the job queue, workflow runs, and job definitions through the existing runledger-postgres admin read APIs.

The dashboard includes continuation volume over 24 hours (Cont 24h), active continued jobs (Cont now), maximum active run depth (Max run), and a total active-continuation KPI. A selected REQUEUED event shows its reason and, for handler continuation, the next run number/time and exact microsecond delay. A selected successful-replay ENQUEUED event shows its source job/run, request key, and reason.

By default it uses an unfiltered global admin scope, so rows from all organizations and rows without an organization are visible. Passing no organization filter is not the same as filtering stored rows to organization_id IS NULL. Pass --org <uuid> at startup, or press o at runtime, to scope to one organization.

export DATABASE_URL=postgres://user:pass@localhost/runledger
cargo run -p runledger-tui

# optional org scope
cargo run -p runledger-tui -- --org 00000000-0000-0000-0000-000000000001

DATABASE_URL must point at a database with the Runledger schema already migrated. The binary runs ensure_schema_compatible_after_idempotency_cutover on startup unless --skip-schema-check is set.

Keys: 1–4 or Tab switch screens · Shift+Tab moves backward · j/k or Up/Down move selection · g/G jump to first/last row · PgUp/PgDn page selection · Enter/l open job/workflow detail · h/Esc go back · [/] or Left/Right switch job-detail panes · / searches the current table · t edits the job/workflow type filter · w edits the workflow type filter from the workflows screen · f cycles queue status filters · c clears contextual filters · v toggles payload wrapping · R toggles raw/pretty payload mode · y copies the selected ID · p pauses auto-refresh · : opens the command palette · r/. refresh · o edits org scope · ? help · q quit.

Configuration

runledger-runtime reads worker settings from the environment via JobsConfig::from_env() (see runledger-runtime/src/config.rs):

Variable	Purpose
JOBS_WORKER_ID	Worker identity; blank falls back to worker-<uuidv7>
JOBS_POLL_INTERVAL_MS	Queue poll interval
JOBS_CLAIM_BATCH_SIZE	Jobs claimed per poll
JOBS_LEASE_TTL_SECONDS	Lease duration; clamped to at least 10
JOBS_MAX_GLOBAL_CONCURRENCY	Max concurrent handler executions
JOBS_REAPER_INTERVAL_SECONDS	Reaper sweep interval
JOBS_SCHEDULE_POLL_INTERVAL_SECONDS	Schedule materialization interval
JOBS_REAPER_RETRY_DELAY_MS	Delay before reaped jobs become claimable

Interval and concurrency values are clamped to safe minimums. JobsConfig::from_env() produces a valid config; if you construct JobsConfig directly, call validate() before starting runtime loops. Supervisor builders reject invalid configs with RuntimeError::InvalidJobsConfig, and low-level loops can return RuntimeLoopExit::InvalidConfig.

Database schema and migrations

The schema is limited to Runledger-owned objects:

• Queue and lifecycle: job_definitions, job_queue, job_attempts, job_events, job_dead_letters, job_schedules, job_execution_resource_claims
• Workflow orchestration: workflow_runs, workflow_steps, workflow_step_dependencies, workflow_run_mutations, workflow_active_claims, workflow_recoveries
• Operational support: job_logs, job_runtime_configs, job_replays
• Derived views: job_metrics_rollup, job_continuation_metrics_rollup

Notable features: idempotent queueing via idempotency_key, cron-backed schedule materialization, workflow DAG execution with dependency counters, external gates via WAITING_FOR_EXTERNAL, append-only workflow mutation tracking, handler continuation and retry audit data, active-workflow and execution-resource claims, immutable replay/recovery lineage, and panic-aware metrics rollups.

A few columns — organization_id, created_by_user_id, updated_by_user_id — are kept for integration flexibility but carry no foreign keys; Runledger treats them as opaque UUIDs. Add referential integrity in your own schema layer if you need it.

Migration set

Migrations live in migrations/ as a flattened baseline plus forward migrations:

• 202603280001_runledger_baseline — the standalone schema baseline (helper functions, queue tables, workflow DAG tables, logs, runtime configs, workflow mutations, external gates, panic-aware attempt outcomes, metrics rollup view).
• 202604100001_runledger_migration_history — creates runledger_migration_history and records the baseline and history-table versions.
• 202605180001_add_enqueue_request_snapshots — adds enqueue_request snapshots to job_queue and workflow_runs so keyed enqueue retries compare the original request instead of mutable runtime state.
• 202605220001_enforce_enqueue_request_snapshots — blocks new keyed rows without snapshots; startup validation rejects pre-cutover legacy rows.
• 202606030001_workflow_results — adds job/step output storage and workflow result handles. Absent result steps are omitted from canonical workflow idempotency snapshots so existing no-result snapshots keep matching.
• 202607190001_job_replays_and_continuation_metrics — adds durable successful job replay lineage and a dedicated continuation metrics rollup. SQLx records and checksum-validates this additive migration in _sqlx_migrations; it deliberately does not add a compatibility-fence row to runledger_migration_history, allowing Runledger's filtered released 0.6.0 startup and schema guards to coexist during expand-first rollout and code rollback. This does not make a raw MIGRATOR.run(...) from that exact release tolerate the newer SQLx history row.
• 202607250001_harden_continuation_metrics_payload_validation — replaces the continuation rollup view so only well-typed, internally consistent handler continuation events contribute to its 24-hour and active-run metrics. This view-only correction also relies on SQLx history without advancing the compatibility-fence history, allowing filtered 0.7.0 startup paths to coexist during the 0.8 rollout.
• 202607280001_workflow_step_handler_continuation — persists the explicit per-step handler-continuation opt-in and prevents external steps from enabling it.
• 202607280002_workflow_active_claims — adds reusable global or organization-scoped active keys, terminal release-pending tracking, and the bounded cleanup index used by the reaper.
• 202607280003_handler_retry_not_before_audit — records requested handler not-before bounds, effective retry times, and whether policy or the handler selected the committed schedule.
• 202607280004_job_execution_resources — adds resource keys to direct jobs and workflow steps, durable lease-fenced resource claims, claim-order indexes, and database triggers that enforce and release exact ownership.
• 202607280005_workflow_recoveries — identifies append mutation records and adds immutable workflow-recovery lineage plus request idempotency.

Every forward migration from 202607190001_job_replays_and_continuation_metrics through 202607280005_workflow_recoveries is recorded and checksum-validated in _sqlx_migrations but deliberately omitted from the custom runledger_migration_history compatibility fence. This lets released filtered startup helpers coexist during the documented expand-first windows; it does not make a raw migrator from an older crate tolerate unknown SQLx history rows.

Treat the flattened baseline as a from-scratch schema definition, not an in-place upgrade from the older multi-file standalone history; apply later forward migrations normally. The workspace-root migrations/ directory is the canonical source for development and review.

Applying or validating the schema

Two supported startup modes:

• migrate_after_idempotency_cutover(&pool) — applies the bundled schema and rejects keyed legacy rows without enqueue snapshots.
• ensure_schema_compatible_after_idempotency_cutover(&pool) — read-only validation that an existing _sqlx_migrations history matches the bundled migrations, with explicit errors for missing history, incompatible history, legacy idempotency rows, or PostgreSQL query/connectivity failures. Externally managed DDL can validate the NOT VALID cutover constraints after this check passes.

For consumers of the published crates:

• runledger_postgres::MIGRATOR embeds the vendored runledger-postgres/migrations/ copy for expert inspection, checksum comparison, and migration-manifest synchronization. Iterating it is supported; directly invoking its raw run or undo methods against a shared pool is not the supported startup path.
• runledger-test-support embeds its own runledger-test-support/migrations/ copy for packaged test harnesses.
• runledger-postgres/build.rs fails local builds if the vendored copy drifts from the canonical workspace-root migrations/ directory.

Call migrate_after_idempotency_cutover to apply migrations, or ensure_schema_compatible_after_idempotency_cutover when DDL is managed externally, before using runledger-postgres or running DB-backed tests. SQLx 0.8 can return early from a raw migration-history rejection without releasing its session advisory lock. A process that unavoidably executes a raw migrator must use a disposable connection or pool and close it after any error rather than retrying with the possibly locked pool.

Release 0.8 requires the complete migration set through 202607280005_workflow_recoveries before any 0.8 runtime loop or persistence API runs. migrate_after_idempotency_cutover may apply it during process startup before those paths begin. In particular, 202607190001_job_replays_and_continuation_metrics is required before replay or continuation-metrics calls, and 202607250001_harden_continuation_metrics_payload_validation supplies the corrected metrics contract. The five 20260728000* migrations supply columns, tables, triggers, and constraints referenced by 0.8 runtime paths.

For a 0.8-to-0.7 code rollback, choose one migration-history strategy only after completing the runtime drain in the activation and rollback runbook:

1. Recommended: leave every 0.8 migration applied and start the 0.7 binary with migrate_after_idempotency_cutover or ensure_schema_compatible_after_idempotency_cutover. These Runledger paths filter SQLx history to migrations embedded in that release. Patch the rollback binary first if it calls raw MIGRATOR.run(...).
2. A raw 0.7 MIGRATOR.run(...) rejects 202607250001_harden_continuation_metrics_payload_validation and the five 20260728000* rows because they are absent from its bundle. SQLx 0.8 may leave that failed session's advisory migration lock held, so close the disposable connection or pool rather than retrying it. If startup cannot be patched, use the 0.8 artifact to revert those six migrations in reverse order before starting 0.7.

The raw down-migration path discards 0.8 state: workflow-recovery lineage and request idempotency, persisted active claims, execution-resource keys and claims, retry-timing audit columns, and workflow-step continuation opt-ins. The workflow/job rows themselves remain, which can leave recovery-created runs without lineage and erase resource constraints from retained work. Reverting further to a pre-0.7 raw bundle also requires reverting 202607190001_job_replays_and_continuation_metrics; that additionally deletes relational successful-replay lineage and replay-request idempotency while leaving replay-created queue rows and their lineage-bearing ENQUEUED events. Use either destructive path only with explicit acceptance of those losses.

Enqueue-request snapshot cutover

Apply the bundled migrations, then run one of the startup APIs. If it returns SchemaCompatibilityError::LegacyIdempotencySnapshotsMissing:

1. Inspect legacy rows with the idx_job_queue_missing_enqueue_request_snapshot and idx_workflow_runs_missing_enqueue_request_snapshot partial indexes.
2. Remediate or drain those keyed rows, then retry startup.

Prefer natural drain, or clearing the stale idempotency_key where retry identity no longer matters. Only backfill enqueue_request when you have the original canonical enqueue request — never reconstruct it from mutable live queue/workflow state; keyed rows created before snapshots existed cannot be safely reconstructed, and keyed retries against them return dedicated conflict errors. migrate_after_idempotency_cutover validates the cutover constraints once no legacy rows remain; that first validation scans job_queue and workflow_runs and may briefly delay startup on large tables without blocking ordinary DML. The cutover migration also builds helper indexes — on large tables, apply it during a maintenance window appropriate for your write volume.

Operational notes

Stable behaviors worth knowing when integrating against runledger-postgres:

• Client-safe errors. QueryError's Display and Debug omit internal database context and are safe for public surfaces; use QueryError::internal_message() for server-side diagnostics. Branch on QueryError::kind() only for the small, compile-checked set of cross-crate runtime policy decisions represented by QueryErrorKind; application and protocol handling should normally use the stable string returned by QueryError::code().
• Lease ownership. Worker lifecycle updates reject expired leases with the stable job.lease_owner_mismatch code, even when the lease was lost by time rather than to another worker. Once lease_expires_at passes there is no owner grace period for heartbeat/progress/success/failure/continuation writes. Release 0.9.0 adds JobLeaseIdentity plus heartbeat_job_for_lease, update_job_progress_for_lease, and success/failure/continuation _for_lease variants for custom runtimes. Reuse one identity derived from the claimed row and worker ID so those four lease fences cannot be mixed across jobs; the positional functions remain compatibility wrappers.
• Transactional enqueue state. Use enqueue_job_with_outcome_tx when the caller needs the job ID together with its locked status, run_number, and Inserted/Existing disposition. That API takes a mutation-ready lock on an existing keyed row. enqueue_job_tx remains the UUID-only compatibility API and retains key-share concurrency between identical keyed enqueues while composing safely with same-transaction compare-and-requeue.
• Compare-and-requeue. Use pool-owning compare_and_requeue_job for a standalone recovery or compare_and_requeue_job_tx when recovery must compose atomically with application writes. Build an exact request from an observed JobQueueRecord with CompareAndRequeueJob::from_observed_job, or provide the expectations explicitly. JobScope::Global matches only a global row, JobScope::Organization(id) matches only that tenant, and RequeueableJobStatus deliberately cannot represent SUCCEEDED. Stale status or run expectations and missing rows are returned as no-mutation outcomes without locking a live worker row. The caller transaction must use READ COMMITTED; other isolation levels return job.compare_and_requeue_unsupported_isolation before lookup. Canceling a leased job preserves its original expiry as a quiescence marker; recovery returns CancellationNotQuiesced { retry_after, .. } until that marker passes, preventing a healthy canceled handler from overlapping the replacement run. Every request must choose JobRequeueStatePolicy::PreserveProgressAndCheckpoint to resume from committed state or ResetProgressAndCheckpoint to restart from scratch; the selected policy is recorded in the REQUEUED event. The older pool-owning requeue_job API is deprecated for 0.6 compatibility; its organization_id: None is an unconstrained lookup, not JobScope::Global, and its behavior corresponds to ResetProgressAndCheckpoint. Migrate every compatibility caller deliberately and handle NotFound, ExpectationMismatch, and CancellationNotQuiesced as no-mutation outcomes. The typed API intentionally does not replay SUCCEEDED jobs in place; use the separate successful-replay API described below. When a canceled handler's retained lease is still active, the compatibility API returns the conflict code job.cancellation_not_quiesced so compatibility callers know the recovery may be retried after lease expiry.
• Successful replay. compare_and_replay_succeeded_job creates an idempotent fresh job from an exactly scoped successful direct-job run; compare_and_replay_succeeded_job_tx composes the same operation with a caller-owned READ COMMITTED transaction. The required replay_request_key identifies one replay action and the required reason is audited. Keys must be non-blank and at most 512 bytes, and reasons must be non-blank. Reusing the same source run, key, and reason returns the existing replay; reusing the key with a different reason returns job.replay_idempotency_conflict. The source row and output remain unchanged. The replay starts at run one with a new ID, copied payload/effective execution settings including any execution resource, no copied progress/checkpoint/output/original idempotency key, and lineage in job_replays plus its ENQUEUED event. Inspect CompareAndReplaySucceededJobOutcome::Replayed.replay.disposition to distinguish insertion from an idempotent retry; ExpectationMismatch and NotFound do not create a job. Queue retention cannot delete only the replay row while its source remains, because that would erase the idempotency guard. Deleting the source cascades its lineage; a single retention statement may delete both source and replay rows together.
• Success stage. complete_job_success persists JobStage::Completed; any other success stage is rejected as a caller error.
• Workflow release conflicts. Workflow-backed job completion waits for an in-flight workflow cancellation to commit or roll back instead of returning a transient workflow.release_conflict. Append and external-step release paths may still return workflow.release_conflict while cancellation holds the exclusive release lock.
• Workflow-managed jobs. Jobs created for workflow steps cannot be requeued or replayed directly with these job-level APIs; that returns job.workflow_requeue_not_supported so the workflow DAG cannot be bypassed. Use workflow cancellation, external completion, or append APIs for workflow-level recovery.
• Stable error codes. Conflicts such as workflow.append_conflicting_retry are conflict-category errors; branch on the stable code rather than the broad category.
• Isolation. Release-sensitive workflow operations, workflow append mutations, and keyed enqueue retries require PostgreSQL READ COMMITTED semantics. READ UNCOMMITTED is accepted because PostgreSQL implements it as read committed.

Migration note for 0.3.x: catalog sync error variants that carry persistence errors now box them as Box<runledger_postgres::Error> to keep Result<_, CatalogError> and Result<_, JobDefinitionCatalogSyncError> small. Downstream code matching those variants should dereference the boxed source before matching the inner persistence error.

PostgreSQL requirements

Runledger requires PostgreSQL 18 or later. PostgreSQL 18 is the authoritative baseline for production support, diagnostics, reproductions, DB-backed tests, migration verification, and SQLx metadata. In particular:

• Native uuidv7() support from PostgreSQL 18+ is required; adding an equivalent function to an older server does not make that server supported.
• Transactional DDL must support the baseline migration as written.
• The target database must be migrated before runtime code uses it.

Working in this repository

Build and test

cargo check
cargo test --workspace --no-run
cargo test -p runledger-core
cargo test -p runledger-postgres
cargo test -p runledger-runtime
cargo check -p runledger-tui

Tests fall into two categories:

• Pure Rust unit tests — no PostgreSQL required.
• DB-backed tests — use runledger-test-support and testcontainers. They start a shared PostgreSQL container, create an isolated ephemeral database per test, and apply the local Runledger migrations.

The packaged external-consumer smoke test packages runledger-core, runledger-test-support, runledger-postgres, and runledger-runtime, extracts the .crate archives, builds a standalone host crate against the packaged manifests via [patch.crates-io] and its checked-in lockfile, then runs migrations, starts the supervisor, enqueues jobs, and asserts terminal states:

./scripts/run-external-consumer-smoke.sh

The default test image is postgres:18. RUNLEDGER_TEST_PG_IMAGE may select a different PostgreSQL 18+ image for an explicit environment or compatibility test, but an override does not change the supported baseline. Results from an older major version are provisional until reproduced on PostgreSQL 18.

export RUNLEDGER_TEST_PG_IMAGE=postgres:18

SQLx offline mode

The repo uses sqlx::query! and friends extensively, and builds offline:

• .cargo/config.toml sets SQLX_OFFLINE=true.
• The workspace-root .sqlx/ directory is the source cache, generated by cargo sqlx prepare --workspace.
• Each publishable crate that uses checked macros also carries its own .sqlx/ so cargo publish can verify the packaged tarball in isolation.

If you change SQL or the schema, refresh the cache before committing:

1. Bring up a PostgreSQL 18 database with the current migrations applied.
2. Point DATABASE_URL at it.
3. Run ./scripts/refresh-sqlx-cache.sh.

The script regenerates the root .sqlx/, syncs it into runledger-postgres/.sqlx/ and runledger-runtime/.sqlx/, syncs the root migrations/ into runledger-postgres/migrations/, runs cargo check --workspace, and confirms the publishable tarballs include their per-crate cache. Do not update only the root .sqlx/ — cargo publish verifies each crate from its packaged tarball. If the cache and schema drift apart, cargo check fails during macro expansion.

Development conventions

• Keep contracts in runledger-core, runtime orchestration in runledger-runtime, and SQL/state-machine logic in runledger-postgres.
• Treat the migration set as the canonical persisted contract for queue and workflow behavior.
• When schema semantics change, update Rust types, SQL, tests, and .sqlx metadata together.
• The repo compiles offline, but DB-backed behavior still needs PostgreSQL 18+ with the current migrations applied.

Releasing

Prepare a release:

./scripts/prepare-release.sh 0.9.1

The preparation script starts from a clean working tree or resumes an existing generated release diff whose manifests are already at the requested version. It rejects changes outside the files it generates. The script bumps publishable crate and root workspace dependency versions, refreshes the root and standalone smoke lockfiles plus SQLx offline metadata, runs workspace tests and the locked packaged smoke test, dry-runs runledger-core, packages the library crates, and build-verifies the packaged runledger-tui binary. It also verifies that every crate archive contains the repository license. If publishing manually, run ./scripts/refresh-sqlx-cache.sh before publishing runledger-postgres or runledger-runtime and commit any resulting .sqlx/ changes.

After reviewing and committing the prepared diff:

./scripts/publish-release.sh 0.9.1

Before publishing any crate, the publish script confirms that the release tag is absent locally and remotely, requires the same-named remote branch to point at the exact local commit, and verifies that commit's completed GitHub Actions CI run and every job succeeded. It then dry-runs the branch and tag push, publishes crates in dependency order, dry-runs each once its workspace dependencies are indexed, creates a v0.9.1 tag, and atomically pushes the current branch and tag. The publication preflight requires an authenticated GitHub CLI. Set PUBLISH_REMOTE to override the git remote for the final push.

Observable contract changes to call out in release notes for this line:

• WorkflowStepEnqueue::execution() provides a typed view of existing job-backed and external workflow-step execution settings.
• Existing workflow builders and individual execution-setting getters remain source compatible, and persisted workflow snapshots retain their existing shape.
• The release adds no migrations beyond the schema required by 0.8.0; the remaining changes are internal module, validation, transaction-phase, TUI, test, packaging, and release-tooling improvements.

See CHANGELOG.md for the full history.

Repository layout

.
├── Cargo.toml                # workspace manifest
├── README.md
├── LICENSE
├── CHANGELOG.md
├── llms.txt                  # prompt-facing summary
├── migrations/               # canonical schema source
├── docs/                     # downstream agent guide and notes
├── scripts/                  # release, SQLx cache, and smoke-test scripts
├── smoke/                    # external-consumer smoke test crate
├── runledger-core/
├── runledger-postgres/
├── runledger-runtime/
├── runledger-tui/
└── runledger-test-support/

License

The crates are published under the MIT license, as declared in each crate's Cargo.toml. See LICENSE for the repository license text.