later

A distributed background job manager and runner for Rust.

Try it

From the workspace root, run the interactive example:

make run-example

Press Enter to use SQLite and the default six workers. The command starts the demo, job dashboard, Prometheus, and Grafana, then prints their local URLs. Use the demo page to enqueue jobs, then watch the job dashboard and Grafana.

The job dashboard shows live stage counts, worker activity, throughput, recent jobs, retries, and sequential topic partitions.

Grafana shows enqueue, completion, retry, worker, and latency metrics collected by Prometheus from the same example.

Set up

Look at the documentations for details. In general the one time setup involves:

• Import later and required dependencies. Enable SQLite or Postgres.
• Define some types to use as a payload to the background jobs
• Generate the stub
• Use the generated code to bootstrap the background job server

Features

Backends

Enable one SQL feature and pass its backend to Config::backend:

• postgres supports services running on several hosts.
• sqlite supports several workers and processes sharing one local file.

Create SQLite storage with later::storage::Sqlite::new("sqlite://later.db").

Delivery

SQLite and Postgres deliver jobs through their own database queue. Start several servers with the same backend namespace to distribute jobs between their workers.

Use Postgres when the servers run on several hosts. SQLite can be shared by several server processes only when they run on the same host and use the same database file.

Both SQL backends accept an existing SQLx pool. Keep a clone of the backend and call enqueue_in with the application's transaction when application data and the job must commit together.

Fire and forget jobs

Fire and forget jobs are executed only once and executed by an available worker almost immediately.

Continuations

One or many jobs are chained together to create an workflow. Child jobs are executed only when parent job has been finished.

Delayed jobs

Just like fire and forget jobs that starts after a certain interval.

Recurring jobs

(wip)

Run recurring jobs based on cron schedule.

• To fix: to delete recurring job.

Dashboard

The dashboard lets you view scheduled, running, retried, completed, and failed jobs. Enable it with the dashboard feature alongside a storage and transport:

later = { version = "0.0.28", features = ["sqlite", "dashboard"] }

Mount a route in the host application and pass its path and raw query string to get_dashboard. The host application owns authentication and access control. Dashboard responses do not add a wildcard CORS header.

The dashboard stores job metadata but does not store payload bytes. Metadata is kept for 24 hours after a completed job expires. It starts tracking changes only after the dashboard is enabled, so jobs created earlier are not rebuilt. The dashboard HTML, JavaScript, and CSS are bundled with the crate.

View the route example in examples/dashboard.rs.

Minimal examples

Start the local services with make init from the workspace root. Run eval "$(make env)" in the terminal where you will run an example. Each example enqueues a few jobs and keeps the server running until Ctrl-C:

cargo run -p later --example sqlite --features sqlite
cargo run -p later --example postgres --features postgres
cargo run -p later --example dashboard --features sqlite,dashboard

The dashboard example asks the operating system for a free local port and prints the frontend and dashboard URLs. The frontend can enqueue single jobs, continuation chains, and jobs that fail once before retrying. Run make down when you're done.