Runledger

Runledger is a standalone Rust workspace for durable job execution and workflow orchestration on PostgreSQL.

This repository was extracted from a larger application and scoped down to the Runledger-specific crates, migrations, and test utilities needed to build and evolve the job system independently.

Workspace

The workspace contains four crates:

• runledger-core Storage-agnostic contracts: job handler traits, runtime types, statuses, identifiers, and workflow enqueue/build validation.
• runledger-postgres SQLx-backed PostgreSQL persistence for the queue, job lifecycle, schedules, workflow DAG state machine, runtime configs, logs, and admin reads/mutations.
• runledger-runtime Async worker, scheduler, and reaper loops plus runtime configuration and handler registry.
• runledger-test-support Published test utilities for ephemeral PostgreSQL databases and scoped environment-variable overrides.

The root workspace manifest is Cargo.toml.

What This Repo Includes

• Rust crates for the Runledger contracts, runtime, and PostgreSQL persistence layer
• A Runledger-only SQL migration history in migrations
• Vendored copies of those migrations in runledger-postgres/migrations and runledger-test-support/migrations so packaged crates can apply schemas without relying on repo-relative paths
• Local test support for DB-backed tests using testcontainers
• SQLx offline metadata in .sqlx/ so the macro-based queries compile without a live database during normal builds

What This Repo Does Not Include

• Application-specific handlers
• API servers, CLIs, or binaries
• Non-Runledger product schema from the original application
• Domain models owned by a larger app

You are expected to embed these crates inside your own service and supply:

• concrete job handlers
• process bootstrapping
• database provisioning
• application-level auth/admin surfaces

Crate Responsibilities

runledger-core

Use runledger-core for the public contracts shared across the rest of the workspace:

• JobHandler and JobHandlerRegistry
• JobContext, JobProgress, and JobFailure
• job status and event enums
• workflow enqueue builders and DAG validation

This crate intentionally has no persistence or async loop logic.

runledger-postgres

Use runledger-postgres when you need durable state in PostgreSQL.

Key capabilities:

• enqueue, claim, heartbeat, retry, succeed, cancel, dead-letter, and requeue jobs
• materialize and update cron schedules
• persist job logs and runtime configs
• create, read, mutate, and advance workflow runs and steps
• query operator/admin views over queue and workflow state

The crate assumes the matching Runledger schema has already been migrated into the target database.

For consumer setup there are two supported modes:

• call runledger_postgres::migrate_after_idempotency_cutover(&pool) to apply the bundled schema during startup and reject keyed legacy rows without enqueue snapshots
• call runledger_postgres::ensure_schema_compatible_after_idempotency_cutover(&pool) to perform a read-only validation that an existing _sqlx_migrations history matches the bundled migrations, with explicit errors for missing history, incompatible history, legacy idempotency rows, or PostgreSQL query/connectivity failures

Operational API notes:

• QueryError::Display and Debug are safe for public surfaces and omit internal database context; use QueryError::internal_message() for server-side diagnostics.
• Worker lifecycle updates reject expired leases with the stable job.lease_owner_mismatch code, even when the lease was lost by time rather than by another worker; once lease_expires_at has passed there is no owner grace period.
• complete_job_success persists JobStage::Completed; passing any other success stage is rejected as a caller error.
• Workflow-backed job completion waits for an in-flight workflow cancellation to commit or roll back instead of returning a transient workflow.release_conflict; append and external-step release paths may still return workflow.release_conflict while cancellation owns the exclusive release lock.
• Retry conflicts such as workflow.append_conflicting_retry are reported as conflict-category query errors; clients should prefer stable error codes over broad categories for exact branching.
• Release-sensitive workflow operations, workflow append mutations, and keyed enqueue retries require PostgreSQL READ COMMITTED semantics. PostgreSQL's READ UNCOMMITTED mode is accepted because PostgreSQL implements it as read committed.
• Keyed rows created before enqueue snapshots existed cannot be safely reconstructed. The idempotency cutover rejects keyed job and workflow rows with enqueue_request IS NULL during startup/schema validation, and keyed retries against such rows return dedicated conflict errors instead of falling back to mutable state comparisons.

runledger-runtime

Use runledger-runtime to run the operational loops around the storage layer:

• worker::run_worker_loop
• scheduler::run_scheduler_loop
• reaper::run_reaper_loop
• registry::JobRegistry
• config::JobsConfig

The runtime is generic. It does not know about your application-specific job catalog beyond the handlers you register.

runledger-test-support

This crate provides shared testing utilities for Runledger crates and downstream integration tests.

It provides:

• setup_ephemeral_pool
• teardown_ephemeral_pool
• ScopedEnv

It starts a disposable PostgreSQL container, creates per-test databases, and runs its vendored Runledger migrations against them. It is published so package tests in runledger-postgres can depend on the same harness that workspace tests use.

Database Model

The standalone schema is intentionally limited to Runledger-owned objects.

Major schema areas:

• queue and lifecycle tables job_definitions, job_queue, job_attempts, job_events, job_dead_letters, job_schedules
• workflow orchestration tables workflow_runs, workflow_steps, workflow_step_dependencies, workflow_run_mutations
• operational support tables job_logs, job_runtime_configs
• derived operational view job_metrics_rollup

Notable schema features:

• idempotent queueing via idempotency_key
• cron-backed schedule materialization
• workflow DAG execution with dependency counters
• external workflow gates via WAITING_FOR_EXTERNAL
• append-only workflow mutation tracking
• panic-aware job metrics rollups

Schema Scope Difference From The Original App

This repository no longer ships the original product schema.

A few columns remain for integration flexibility, but their original foreign keys were intentionally removed in the standalone migration set:

• organization_id
• created_by_user_id
• updated_by_user_id

These values are now treated as opaque UUIDs from the perspective of Runledger. If your host application wants referential integrity, it should add that in its own schema layer or wrap these migrations with app-owned extensions.

Migrations

The migration set lives in migrations.

This repo uses a flattened baseline plus forward migrations:

• 202603280001_runledger_baseline creates the standalone Runledger schema baseline, including: helper functions, queue tables, workflow DAG tables, logs, runtime configs, workflow mutations, external workflow gates, panic-aware attempt outcomes, and the final metrics rollup view
• 202604100001_runledger_migration_history creates runledger_migration_history and records the standalone baseline and history-table migration versions
• 202605180001_add_enqueue_request_snapshots adds enqueue_request snapshots to job_queue and workflow_runs so keyed enqueue retries can compare the original request instead of mutable runtime state
• 202605220001_enforce_enqueue_request_snapshots blocks new keyed queue/workflow rows without snapshots while startup validation rejects pre-cutover legacy rows; the application migration API validates the cutover constraints after legacy-row validation passes

The historical standalone migration chain was intentionally collapsed because this repository now targets fresh standalone deployments rather than preserving every intermediate extraction-era cutover step.

If you already created databases from the older multi-file standalone migration history, treat the flattened baseline as a new-from-scratch schema definition, not as an in-place upgrade path. Apply later forward migrations normally.

The workspace-root migration directory remains the canonical schema source for repo development and review.

For consumers using the published crate:

• runledger_postgres::MIGRATOR embeds the vendored runledger-postgres/migrations/ copy
• runledger-test-support embeds its own runledger-test-support/migrations/ copy for packaged test harnesses
• runledger_postgres::migrate_after_idempotency_cutover(&pool) applies those migrations and rejects keyed legacy rows without snapshots
• runledger_postgres::ensure_schema_compatible_after_idempotency_cutover(&pool) validates that an existing _sqlx_migrations history matches them without running DDL and returns Runledger-specific errors for missing history, incompatible history, legacy idempotency rows, or PostgreSQL query/connectivity failures; externally managed DDL can validate the NOT VALID cutover constraints after this check passes
• runledger-postgres/build.rs fails local builds if the vendored crate copy drifts from the canonical workspace-root migrations/ directory

Apply these migrations, or call runledger_postgres::migrate_after_idempotency_cutover(&pool), before using runledger-postgres or running DB-backed tests.

For the enqueue-request snapshot cutover, apply the bundled migrations first, then run either startup API. If it returns SchemaCompatibilityError::LegacyIdempotencySnapshotsMissing, inspect the legacy rows with the idx_job_queue_missing_enqueue_request_snapshot and idx_workflow_runs_missing_enqueue_request_snapshot partial indexes, remediate or drain those keyed rows, and retry startup. Prefer natural drain or clearing the stale idempotency_key where retry identity no longer matters; only backfill enqueue_request when you have the original canonical enqueue request, not from mutable live queue/workflow state. migrate_after_idempotency_cutover validates the cutover constraints once no legacy rows remain; that first validation scans job_queue and workflow_runs and may briefly delay startup on large tables without blocking ordinary DML. The cutover migration also builds helper indexes for locating legacy rows; on large tables, apply it during a maintenance window appropriate for your write volume.

Runtime Configuration

runledger-runtime exposes JobsConfig::from_env() in runledger-runtime/src/config.rs.

Supported environment variables:

• JOBS_WORKER_ID
• JOBS_POLL_INTERVAL_MS
• JOBS_CLAIM_BATCH_SIZE
• JOBS_LEASE_TTL_SECONDS
• JOBS_MAX_GLOBAL_CONCURRENCY
• JOBS_REAPER_INTERVAL_SECONDS
• JOBS_SCHEDULE_POLL_INTERVAL_SECONDS
• JOBS_REAPER_RETRY_DELAY_MS

Default behavior:

• blank JOBS_WORKER_ID falls back to worker-<uuidv7>
• interval and concurrency values are clamped to safe minimums
• lease TTL is clamped to at least 10 seconds

Building

Common commands:

cargo check
cargo test --workspace --no-run
cargo test -p runledger-core
cargo test -p runledger-postgres
cargo test -p runledger-runtime
./scripts/run-external-consumer-smoke.sh

The standalone workspace has been validated with:

cargo check
cargo test --workspace --no-run

SQLx Offline Mode

This repo uses sqlx::query! and related macros extensively.

To keep normal builds self-contained:

• .cargo/config.toml sets SQLX_OFFLINE=true
• the workspace-root .sqlx/ directory is the source cache generated by cargo sqlx prepare --workspace
• each publishable crate that uses SQLx checked macros also carries its own .sqlx/ directory so cargo publish can verify the packaged tarball in isolation

If you change SQL queries or the schema, refresh the cache before committing.

Typical workflow:

1. bring up a PostgreSQL database with the current Runledger migrations applied
2. point DATABASE_URL at that database
3. run ./scripts/refresh-sqlx-cache.sh

What the script does:

• regenerates the workspace root .sqlx/ cache
• syncs that cache into runledger-postgres/.sqlx/ and runledger-runtime/.sqlx/
• syncs the workspace-root migrations/ directory into runledger-postgres/migrations/
• runs cargo check --workspace
• confirms the publishable crate tarballs include their per-crate SQLx cache

Do not update only the workspace root .sqlx/ directory. cargo publish verifies each crate from its packaged tarball, so publishable crates must include their own SQLx cache.

If the cache and schema drift apart, cargo check will fail during macro expansion.

Publishing

Prepare a release with the repository script:

./scripts/prepare-release.sh 0.1.2

The preparation script:

• requires a clean working tree
• bumps publishable crate versions and root workspace dependency versions
• refreshes SQLx offline metadata
• runs workspace tests and the packaged external-consumer smoke test
• runs a publish dry-run for runledger-core and packages the dependent crates locally

Before publishing this release line, call out observable contract changes in release notes: QueryError::Display now returns client-safe messages, expired leases have no owner grace period for heartbeat/progress/success/failure writes, the job.lease_owner_mismatch message now covers time-based loss of ownership, success completion rejects non-Completed stages, workflow-backed job completion waits on in-flight cancellation instead of returning workflow.release_conflict, append/external release can still return workflow.release_conflict, workflow append mutations require read-committed transaction isolation, idempotent enqueue adds new conflict/isolation error codes, and workflow.append_conflicting_retry is now a conflict-category error.

If publishing manually, run ./scripts/refresh-sqlx-cache.sh before publishing runledger-postgres or runledger-runtime and commit any resulting .sqlx/ changes.

After reviewing and committing the prepared diff, publish with:

./scripts/publish-release.sh 0.1.2

The publish script publishes crates in dependency order, dry-runs each crate once its workspace dependencies are indexed, creates a v0.1.2 tag, and pushes the current branch and tag. Set PUBLISH_REMOTE to override the git remote used for the final push.

Testing

There are two main categories of tests:

• pure Rust unit tests these do not require PostgreSQL
• DB-backed tests these use runledger-test-support and testcontainers

The DB-backed tests:

• start a shared PostgreSQL container
• create isolated ephemeral databases per test
• apply the local Runledger migrations

The packaged external-consumer smoke test:

• packages runledger-core, runledger-postgres, and runledger-runtime
• extracts those .crate archives locally
• builds a standalone host crate against the packaged manifests via [patch.crates-io]
• runs migrations, starts worker/scheduler/reaper, enqueues jobs, and asserts terminal states

Run it with:

./scripts/run-external-consumer-smoke.sh

The default test image is postgres:18.

Override it with:

export RUNLEDGER_TEST_PG_IMAGE=postgres:18

The test harness expects the database image to support uuidv7().

PostgreSQL Assumptions

Runledger expects PostgreSQL semantics and features consistent with the migration set and SQLx queries in this repo.

In particular:

• uuidv7() must be available
• transactional DDL behavior must support the baseline migration as written
• the target DB must be migrated before runtime code uses it

Typical Integration Shape

A host application will generally:

1. either call runledger_postgres::migrate_after_idempotency_cutover(&pool) or apply the Runledger migrations with your own deployment tooling and then call runledger_postgres::ensure_schema_compatible_after_idempotency_cutover(&pool)
2. create a shared sqlx::PgPool
3. register concrete handlers in runledger_runtime::registry::JobRegistry
4. start worker, scheduler, and reaper loops with coordinated shutdown
5. call runledger_postgres::jobs::* APIs from its own admin/API surfaces

At a high level:

use runledger_runtime::config::JobsConfig;
use runledger_runtime::registry::JobRegistry;

let pool = /* sqlx PgPool */;
runledger_postgres::migrate_after_idempotency_cutover(&pool).await?;

let mut registry = JobRegistry::new();
// registry.register(MyHandler);

let config = JobsConfig::from_env();
// spawn worker/scheduler/reaper loops with the shared pool and registry

This workspace deliberately stops at the library boundary; it does not prescribe your process model or handler packaging.

Repository Layout

.
├── Cargo.toml
├── README.md
├── migrations/
├── runledger-core/
├── runledger-postgres/
├── runledger-runtime/
└── runledger-test-support/

Development Notes

• Prefer keeping contracts in runledger-core, runtime orchestration in runledger-runtime, and SQL/state-machine logic in runledger-postgres.
• Treat the migration set as the canonical persisted contract for queue and workflow behavior.
• When schema semantics change, update Rust types, SQL, tests, and .sqlx metadata together.
• The repo may compile offline, but DB-backed behavior still needs migration-compatible PostgreSQL for execution.

License

No license file is included in this extraction. Add one at the repository root if this workspace is intended for redistribution or open-source use.