Crate taquba 

A durable, single-process task queue for Rust, backed by object storage.

Taquba persists every job-state transition through SlateDB to an object_store backend (local disk, S3, GCS, Azure Blob, MinIO, etc.) so the queue survives process restarts, node loss and ephemeral disks.

When Taquba fits

• A single-binary service that needs durable background jobs without operating a queue server.
• Edge or ephemeral compute where the local disk is gone after each invocation but the bucket persists.
• Low-to-moderate-throughput workloads where cheap per-PUT pricing on object storage beats running a database or broker.

When Taquba does not fit

If you need a worker fleet spread across multiple machines.

Quick start

use std::sync::Arc;
use std::time::Duration;
use taquba::{Queue, object_store::memory::InMemory};

let q = Queue::open(Arc::new(InMemory::new()), "demo").await?;

q.enqueue("email", b"alice@example.com".to_vec()).await?;

if let Some(job) = q.claim("email", Duration::from_secs(30)).await? {
    // ... do the work ...
    q.ack(&job).await?;
}

q.close().await

Job lifecycle

pending → claimed → done
              ↘
           failed → (backoff → pending | dead-letter)

• At-least-once delivery: workers must be idempotent.
• Lease-based claims: a background reaper requeues abandoned jobs.
• Exponential retry backoff via the scheduled key space (configurable per queue, see QueueConfig).
• Bounded dead-letter retention with paginated inspection.

Coordinating with caller state

Queue::enqueue_with_kv enqueues a job and applies a set of writes to a caller-owned KV namespace in a single transaction, so a downstream crate can keep its own durable coordination state (status markers, dedup records, pointers to externally-stored blobs) consistent with the queue across crashes. Queue::kv_get and Queue::kv_delete read and clean up those entries.

Caller keys live under a reserved usr: prefix internally so they cannot collide with Taquba’s own layout. Per-value size is capped at MAX_KV_VALUE_SIZE; the namespace is sized for coordination state, not bulk payload. Store large blobs in the underlying object store under a content-addressed key and put only the pointer in KV.

The namespace is mutated only as a side effect of queue operations; there is no standalone kv_put. To create or update an entry, include it in the kv_writes map of an Queue::enqueue_with_kv call (which makes the write atomic with the enqueue). Queue::kv_delete is the one standalone primitive, for terminal cleanup of entries whose related queue op has already completed.

Background tasks

Queue::open spawns two background tokio tasks for the lifetime of the handle:

• Reaper - re-queues jobs whose lease expired and runs the done / dead-letter retention sweeps (interval: OpenOptions::reaper_interval).
• Scheduler - promotes scheduled jobs whose run_at has passed (interval: OpenOptions::scheduler_interval).

Call Queue::close for a clean shutdown; it stops both tasks and flushes the underlying SlateDB instance.

Cargo features

No backend is enabled by default: the in-memory and local-disk stores work without any feature. Pick exactly one for production:

cargo add taquba --features aws    # S3 / MinIO
cargo add taquba --features gcp    # Google Cloud Storage
cargo add taquba --features azure  # Azure Blob

Re-exports

pub use worker::PermanentFailure;

pub use worker::Worker;

pub use worker::WorkerError;

pub use worker::run_worker;

pub use worker::run_worker_concurrent;

pub use slatedb::object_store;

Modules

worker

Worker-loop primitives: the worker::Worker trait, plus the worker::run_worker / worker::run_worker_concurrent drivers that own the claim -> process -> ack/nack lifecycle and graceful shutdown.

Structs

EnqueueOptions

Per-call overrides for Queue::enqueue_with.

JobRecord

A single job stored in a Taquba queue.

OpenOptions

Configuration for opening a Queue instance.

Queue

A durable task queue backed by object storage.

QueueConfig

Configuration applied to a specific queue (or used as the default for all queues).

QueueStats

A snapshot of job counts for a single queue.

Enums

CancelOutcome

Outcome of Queue::cancel, reflecting which lifecycle branch the job was in.

EnqueueResult

Outcome of Queue::enqueue_with_kv.

Error

Errors returned by Taquba’s public API.

JobStatus

The lifecycle state of a JobRecord.

WaitOutcome

Outcome of Queue::wait_for_completion.

Constants

MAX_KV_VALUE_SIZE

Maximum size of a single value in the user KV namespace.

PRIORITY_HIGH

High-priority bucket. Jobs at this priority are dequeued before normal and low.

PRIORITY_LOW

Low-priority bucket. Jobs at this priority are dequeued after high and normal.

PRIORITY_NORMAL

Default priority. FIFO ordering is preserved within the same priority level.

Type Aliases

Result

Convenience alias for Result<T, Error> returned throughout the crate.