Crate backfill 

Backfill

A high-performance, PostgreSQL-backed async priority queue system for Rust applications. Built on top of Graphile Worker for reliability and performance.

This library provides:

• Durable job queues with PostgreSQL backend
• Priority-based scheduling with configurable priority levels
• Parallel execution by default - jobs run concurrently across all workers
• Serial queues when you need ordering or rate limiting
• Exponential backoff with jitter to prevent thundering herds
• Flexible retry policies (fast, aggressive, conservative, or custom)
• Dead letter queue handling for failed jobs
• Type-safe job handlers using Rust’s type system
• Low-latency execution via PostgreSQL LISTEN/NOTIFY

Quick Start

use backfill::{BackfillClient, enqueue_critical, enqueue_fast_with_retries, JobSpec, RetryPolicy};
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct SendEmailJob {
    recipient: String,
    subject: String,
    body: String,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let database_url = "postgresql://user:password@localhost/mydb";
    let client = BackfillClient::new(database_url).await?;

    // Enqueue a critical job with aggressive retry policy (12 attempts)
    enqueue_critical(
        &client,
        "send_email",
        &SendEmailJob {
            recipient: "user@example.com".to_string(),
            subject: "Welcome!".to_string(),
            body: "Thanks for signing up.".to_string(),
        },
        Some("welcome_email_123".to_string()),
    ).await?;

    // Or use fast retries for quick turnaround (3 attempts, 100ms-30s)
    enqueue_fast_with_retries(
        &client,
        "send_notification",
        &SendEmailJob {
            recipient: "admin@example.com".to_string(),
            subject: "Notification".to_string(),
            body: "Quick notification with fast retries.".to_string(),
        },
        Some("quick_notification".to_string()),
    ).await?;

    Ok(())
}

Modules

cleanup

Cleanup utilities for maintaining worker queue health

metrics

Metrics emission for observability

Structs

AfterJobRun

AfterJobRunContext

BackfillClient

High-level client for the backfill job queue system.

BeforeJobRun

BeforeJobRunContext

BeforeJobSchedule

BeforeJobScheduleContext

CronJobScheduled

CronJobScheduledContext

CronTick

CronTickContext

CrontabParseError

DlqCleanupPlugin

Plugin that cleans up DLQ entries when jobs with matching job_keys complete.

DlqFilter

Filter parameters for querying DLQ jobs.

DlqJob

A job that has been moved to the dead letter queue after failing permanently.

DlqJobList

Paginated list of DLQ jobs.

DlqStats

Statistics about the dead letter queue.

HookRegistry

Job

Job extends DbJob with an additional task_identifier field.

JobComplete

JobCompleteContext

JobFail

JobFailContext

JobFetch

JobFetchContext

JobPermanentlyFail

JobPermanentlyFailContext

JobSpec

Configuration for job scheduling and execution.

JobStart

JobStartContext

Priority

Priority levels for jobs in the backfill system.

QueueConfig

Configuration for a worker queue

RetryPolicy

Configuration for exponential backoff retry policy.

WorkerConfig

Flexible worker configuration for integration with existing applications

WorkerContext

Context provided to task handlers when processing a job.

WorkerInit

WorkerInitContext

WorkerOptions

Configuration options for initializing a Graphile Worker instance.

WorkerOptionsBuilder

Cloneable wrapper around WorkerOptions configuration

WorkerRunner

Reusable worker runner for flexible integration patterns

WorkerRunnerBuilder

Builder for WorkerRunner that allows configuring job types and plugins

WorkerShutdown

WorkerShutdownContext

WorkerStart

WorkerStartContext

Enums

BackfillError

Core errors for the Backfill library

EnqueueOutcome

Outcome of an enqueue operation.

HookResult

JobKeyMode

JobScheduleResult

Queue

Queue configuration for job execution.

ShutdownReason

WorkerError

Worker-specific errors that can classify whether jobs should be retried

Constants

DEFAULT_STALE_JOB_LOCK_TIMEOUT

Default timeout for considering a job lock stale.

DEFAULT_STALE_LOCK_CLEANUP_INTERVAL

Default interval for periodic stale lock cleanup.

DEFAULT_STALE_QUEUE_LOCK_TIMEOUT

Default timeout for considering a queue lock stale.

Traits

IntoTaskHandlerResult

Trait for converting task handler return types into a standardized Result.

Plugin

TaskHandler

Core trait for defining task handlers in Graphile Worker.

Functions

enqueue_bulk

Enqueue a normal-priority job for parallel execution.

enqueue_bulk_with_retries

Enqueue a bulk job with conservative exponential backoff retries.

enqueue_critical

Enqueue a critical job with aggressive exponential backoff retries.

enqueue_emergency

Enqueue an emergency priority job.

enqueue_fast

Enqueue a high-priority job for parallel execution.

enqueue_fast_with_retries

Enqueue a high-priority job with fast exponential backoff retries.

enqueue_serial

Enqueue a job for serial execution within a named queue.

parse_crontab

Parse a crontab definition into a Vec of crontab