Crate disk_backed_queue 

disk-backed-queue

A robust, crash-resistant queue implementation that persists all data to disk using SQLite. Provides an mpsc-like channel API while ensuring messages survive application restarts and system failures.

Features

• Zero Message Loss: All messages are persisted to SQLite before acknowledgment
• Crash Recovery: Messages survive application restarts and system crashes
• MPSC-like API: Familiar channel-based interface compatible with async Rust
• Dead Letter Queue: Corrupted messages automatically moved to separate database
• Batch Operations: High-performance bulk send/receive (460x faster than single operations)
• Backpressure Support: Optional queue size limits prevent unbounded growth

Quick Start

use disk_backed_queue::disk_backed_channel;
use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize, Debug)]
struct MyMessage {
    id: u64,
    content: String,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a disk-backed channel
    let (tx, mut rx) = disk_backed_channel::<MyMessage, _>(
        "my_queue.db",
        "messages".to_string(),
        None  // No size limit
    ).await?;

    // Send messages
    tx.send(MyMessage {
        id: 1,
        content: "Hello, world!".to_string(),
    }).await?;

    // Receive messages
    if let Some(msg) = rx.recv().await? {
        println!("Received: {:?}", msg);
    }

    Ok(())
}

Use Cases

Perfect for scenarios where message loss is unacceptable:

• Message queuing during database outages
• Event sourcing and audit logs
• Job queues that survive restarts
• Data pipelines requiring guaranteed delivery

Performance

Batch operations are approximately 50x faster than single operations due to reduced fsync overhead. For high-throughput scenarios, use DiskBackedSender::send_batch and DiskBackedReceiver::recv_batch.

Run cargo run --example throughput_demo --release to see performance on your system.

Structs

DiskBackedQueue

A disk-backed queue that persists all messages to SQLite.

DiskBackedReceiver

The receiving half of a disk-backed channel.

DiskBackedSender

The sending half of a disk-backed channel.

Enums

DiskQueueError

DurabilityLevel

Durability configuration for SQLite synchronous mode

Functions

disk_backed_channel

Create a disk-backed channel with default durability (FULL).

disk_backed_channel_with_durability

Create a disk-backed channel with custom durability level.

Type Aliases

Result