rust-job-queue-api-worker-system

Copyright 2026 infinityabundance

A compact, production-shaped Rust job queue: Axum HTTP API, Tokio workers, Postgres SELECT ... FOR UPDATE SKIP LOCKED dequeue, retries with decorrelated jitter, idempotency, cooperative cancellation, OpenAPI, Prometheus metrics, Docker Compose, and reproducible tests/benchmarks.

Single Cargo package, two feature flags, two service binaries, and a one-shot migration runner.

What this demonstrates

Evidence

• cargo test --all-features --locked: 46 tests in the current suite (12 lib unit tests, 33 integration tests, 1 doctest). Integration tests require Docker because they use testcontainers Postgres.
• CI workflow: ci.yml.
• Architecture notes: docs/architecture.md.
• API reference: docs/api.md.
• Operational runbook: docs/runbook.md.
• Trade-offs and omissions: docs/tradeoffs.md.
• Colab smoke/repro notebook: notebooks/rust_job_queue_api_worker_system_colab.ipynb.

Quickstart

docker compose up                       # boots postgres + migrations + api + worker
open http://localhost:8080/docs         # Swagger UI
curl http://localhost:8080/metrics      # API Prometheus metrics
curl http://localhost:9091/metrics      # worker Prometheus metrics
cargo test --all-features --locked      # full test suite; needs Docker

Environment variables are documented in .env.example.

Endpoints

Full endpoint reference: docs/api.md.

Architecture in one paragraph

The core modules (domain, payload, db, queue, retry, ids, error) are always compiled; api and worker are gated behind features of the same names. The API and worker binaries use required-features, so they cannot be built without their dependency sets. A third binary, job-queue-migrate, runs embedded migrations and exits; Docker Compose uses it before starting the API and worker.

The queue lives in one Postgres table (jobs) with a partial index on (run_at) filtered by status IN ('queued','retrying'). Retries are scheduled by setting run_at into the future; the same dequeue query picks them up when run_at <= now(). Cancellation is atomic for queued/retrying jobs and cooperative for running jobs.

Full design, including state and sequence diagrams: docs/architecture.md.

Feature flags

Consumers who only want the queue types and SQL helpers can use:

cargo add rust-job-queue-api-worker-system --no-default-features

CI checks core-only, API-only, and worker-only builds.

Tests

cargo test --all-features --locked

The suite currently has 46 tests across lib unit tests, one doctest, and seven integration binaries:

• queue_ops (19): SKIP LOCKED dequeue, retry math, cancel paths, recovery sweep, DB-layer idempotency.
• api_jobs (5): create/get/list, 404, 422, /health, OpenAPI rendering.
• idempotency (4): two-concurrent same-key, 10-concurrent same-key race, different keys, header precedence over body.
• retry_and_cancel (2): fail-once-then-succeed, always-fail-to-permanent.
• worker_dequeue (1): enqueue -> worker processes -> succeeded.
• cancel_running (1): cancellation observed at a safe point.
• concurrency_two_workers (1): 200 jobs across 8 workers, no duplicate processing observed by the test harness.
• lib unit tests (12): payload validation, retry jitter bounds, deterministic simulator hash.
• doctest (1): crate-level example in src/lib.rs.

All integration tests use testcontainers; each test gets an isolated ephemeral database.

Benchmarks

Two release-supported benches are reproducible against a fresh Postgres container in Docker.

# Single-worker queue-overhead microbench (Criterion).
cargo bench --bench dequeue --features worker

# Concurrent throughput across {1,2,4,8,16} workers and 3 Postgres configs.
./bench/run.sh

Measured results are in bench/RESULTS.md. The numbers are scoped to the recorded hardware and methodology. They are not presented as production throughput claims.

The exploratory head-to-head comparison benchmark is intentionally excluded from the v0.1 crates.io package until its methodology and teardown behavior are corrected.

Example: embed the worker in your own application

DATABASE_URL=postgres://localhost/jobs \
  cargo run --example embed_worker --features worker

See examples/embed_worker.rs for a small example of plugging a custom Executor into WorkerRuntime.

CI

ci.yml runs:

1. cargo fmt --check, clippy with warnings denied, feature-decoupling checks, full tests, release build, docs with warnings denied, cargo package --list --locked, and cargo publish --dry-run --locked.
2. cargo +1.88.0 check --all-features --locked for the declared MSRV.
3. RustSec audit.

Trade-offs and intentional omissions

This artifact is deliberately small. The decisions and "what would change in production" notes live in docs/tradeoffs.md. Start there if you are evaluating engineering judgment.

Tech stack

Rust (MSRV 1.88), Tokio 1, Axum 0.8, sqlx 0.8, Postgres 16, tower-http 0.6, tracing 0.1, utoipa 5 + utoipa-swagger-ui 9, metrics + metrics-exporter-prometheus 0.18, thiserror 2, anyhow 1, testcontainers 0.27, Criterion 0.5.

License

Apache-2.0