RRQ

The orchestrator for RRQ, a distributed job queue that combines Rust reliability with language-flexible workers.

What is RRQ?

RRQ (Reliable Redis Queue) separates the hard parts of distributed job processing—scheduling, retries, locking, timeouts—into a single Rust binary. Your job handlers can be written in Python, TypeScript, or Rust, running as isolated processes managed by the orchestrator.

Why choose RRQ?

• Write handlers in any language - Python, TypeScript, or Rust workers connect via socket protocol
• Redis-native - Atomic operations, predictable semantics, no separate database to manage
• Battle-tested Rust core - The complex distributed systems logic runs in optimized Rust
• Production features included - Retries, DLQ, timeouts, cron scheduling, health checks, distributed tracing

Installation

cargo install rrq

Or add to your Cargo.toml:

[dependencies]
rrq = "0.9"

Quick Start

1. Create configuration (rrq.toml)

[rrq]
redis_dsn = "redis://localhost:6379/0"
default_runner_name = "python"

[rrq.runners.python]
type = "socket"
cmd = ["rrq-runner", "--settings", "myapp.runner:settings"]
tcp_socket = "127.0.0.1:9000"
pool_size = 4
max_in_flight = 10

2. Start the worker

rrq worker run --config rrq.toml

The orchestrator spawns runner processes, polls Redis for jobs, dispatches work, handles retries, and manages the entire lifecycle.

3. Enqueue jobs

Use the rrq-producer crate for Rust, or the Python/TypeScript clients:

use rrq_producer::Producer;

let producer = Producer::new("redis://localhost:6379/0").await?;
let job_id = producer.enqueue("my_handler", json!({"key": "value"})).await?;

CLI Commands

Worker

# Production mode
rrq worker run --config rrq.toml

# Watch mode (restarts on file changes)
rrq worker watch --config rrq.toml --path ./src

# Burst mode (exit when queue empty)
rrq worker run --config rrq.toml --burst

Queues

rrq queue list --config rrq.toml
rrq queue inspect default --config rrq.toml
rrq queue pause default --config rrq.toml
rrq queue resume default --config rrq.toml

Jobs

rrq job get <job-id> --config rrq.toml
rrq job cancel <job-id> --config rrq.toml
rrq job retry <job-id> --config rrq.toml

Dead Letter Queue

rrq dlq list --config rrq.toml
rrq dlq retry-all --config rrq.toml
rrq dlq purge --config rrq.toml

Health

rrq health --config rrq.toml

Configuration Reference

[rrq]
redis_dsn = "redis://localhost:6379/0"    # Required
default_runner_name = "python"
default_job_timeout_seconds = 300
default_max_retries = 3
heartbeat_interval_seconds = 60
runner_shutdown_term_grace_seconds = 5.0
runner_enable_inflight_cancel_hints = false

[rrq.runners.python]
type = "socket"
cmd = ["rrq-runner", "--settings", "myapp.runner:settings"]
tcp_socket = "127.0.0.1:9000"
pool_size = 4           # Runner processes
max_in_flight = 10      # Concurrent jobs per runner
cwd = "/app"            # Working directory (optional)

[rrq.runners.python.env]
PYTHONPATH = "/app"     # Environment variables (optional)

# Cancellation behavior:
# - Pending jobs: deterministic cancellation (`rrq job cancel`)
# - In-flight jobs: shutdown/timeout enforcement is process lifecycle based
#   (Rust orchestrator handles TERM -> grace -> KILL on Unix)
# - Cancel hints to runners are optional and disabled by default

# Cron jobs
[[rrq.cron_jobs]]
function_name = "daily_cleanup"
schedule = "0 0 9 * * *"   # 6-field cron (with seconds)
queue_name = "maintenance"

# Watch mode settings
[rrq.watch]
path = "."
include_patterns = ["*.py", "*.toml"]
ignore_patterns = [".venv/**", "dist/**"]
#
# Optional build steps (useful for compiled handlers/runners):
# - Each entry is an argv array.
# - If any command fails, the worker remains stopped until the next change.
pre_restart_cmds = [["cargo", "build"]]
pre_restart_cwd = "."

Architecture

┌─────────────────────────────────────────────────────────────┐
│                     RRQ Orchestrator                         │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────┐  │
│  │   Worker    │  │   Worker    │  │   Cron Scheduler    │  │
│  │   Loop      │  │   Loop      │  │                     │  │
│  └──────┬──────┘  └──────┬──────┘  └──────────┬──────────┘  │
│         │                │                    │              │
│         └────────────────┼────────────────────┘              │
│                          │                                   │
│                  ┌───────▼───────┐                           │
│                  │  Redis Store  │                           │
│                  └───────┬───────┘                           │
│                          │                                   │
│              ┌───────────┴───────────┐                       │
│              │                       │                       │
│      ┌───────▼───────┐       ┌───────▼───────┐              │
│      │  Runner Pool  │       │  Runner Pool  │              │
│      │   (Python)    │       │    (Node)     │              │
│      └───────────────┘       └───────────────┘              │
└─────────────────────────────────────────────────────────────┘

Environment Variables

Related Crates

Cross-Language Compatibility

The orchestrator works seamlessly with:

• Python workers via rrq
• TypeScript workers via rrq-ts
• Rust workers via rrq-runner

All use the same Redis schema and socket protocol.

License

Apache-2.0