Crate queue_file 

A lightning-fast, transactional, file-based FIFO queue.

This crate is a feature-complete port of the QueueFile class from Tape2 by Square, Inc., extended with a new v2 format that adds per-element sequence numbers, CRC-32 integrity checks, and dual-slot atomic header commits.

Overview

QueueFile stores an ordered sequence of byte blobs in a single file, using a ring-buffer layout for O(1) enqueue and dequeue. The file grows on demand (doubling its size) and shrinks back to QueueFile::with_capacity’s capacity floor when the queue is cleared.

Atomicity guarantee

Every mutation commits by rewriting the file header last. Because the header is small enough that filesystem implementations typically write it atomically, a process crash after data has been written but before the header is committed leaves the queue in its previous consistent state. The stored file-length field additionally allows recovery from a failed expansion: if the file was grown but the copy was not finished, the old length can be recovered from the header.

File formats

Three header formats are supported:

• V2 (dual 56-byte slots, default): created by QueueFile::open and QueueFile::with_capacity. Supports per-element CRC integrity, sequence numbers, and backlink recovery. Files created by older versions are automatically migrated on open.
• V1 (32 bytes): the previous default format, now migrated to V2 on open.
• Legacy (16 bytes): created by QueueFile::open_legacy. Binary-compatible with the original Java QueueFile. Supports files up to i32::MAX bytes only.

Performance notes

• Use QueueFile::add_n to batch multiple elements into a single write.
• Set QueueFile::set_sync_writes to false if durability after every operation is not required (e.g. when building an in-process write-ahead log that controls flushing itself).
• Enable QueueFile::set_skip_write_header_on_add with true together with QueueFile::sync_all to defer header writes across a series of additions.
• Use QueueFile::set_cache_offset_policy to speed up random access through Iter::nth on large queues.

Thread safety

QueueFile is not thread-safe: it is neither Send nor Sync. File I/O operations on a shared object from multiple threads would corrupt the on-disk state and the in-memory queue bookkeeping. Wrap it in a synchronization primitive for cross-thread use, e.g.:

use std::sync::{Mutex, RwLock};

// Safe: `Mutex` provides exclusive access
let qf = Mutex::new(QueueFile::open("queue.qf")?);
qf.lock().unwrap().add(b"data")?;

// Safe: `RwLock` allows concurrent reads with exclusive writes
let qf = RwLock::new(QueueFile::open("queue.qf")?);
let data = qf.read().unwrap().peek()?;

Opening the same queue file from multiple processes simultaneously is not supported and will result in data corruption.

Quick Start

use queue_file::QueueFile;
use std::fs::remove_file;

// Create a new queue file
let path = "example.qf";
let mut qf = QueueFile::open(path).unwrap();

// Add elements
qf.add(b"hello").unwrap();
qf.add(b"world").unwrap();

// Peek at the front element
assert_eq!(qf.peek().unwrap(), Some(b"hello".to_vec()));

// Iterate over all elements
for elem in &qf {
    println!("{}", String::from_utf8_lossy(&elem));
}

// Remove elements
qf.remove().unwrap();
assert_eq!(qf.peek().unwrap(), Some(b"world".to_vec()));

// Clean up
drop(qf);
remove_file(path).ok();

Batch Operations

For better performance when adding multiple elements, use QueueFile::add_n:

use queue_file::QueueFile;
use std::fs::remove_file;

let path = "batch.qf";
let mut qf = QueueFile::open(path).unwrap();

// Add multiple elements in a single batch (more efficient)
qf.add_n(vec![b"a", b"b", b"c"]).unwrap();
assert_eq!(qf.size(), 3);

drop(qf);
remove_file(path).ok();

Random Access with Caching

For frequent access to elements by index, enable the cache:

use queue_file::{QueueFile, OffsetCacheKind};
use std::fs::remove_file;

let path = "cache.qf";
let mut qf = QueueFile::open(path).unwrap();

// Enable linear caching for faster nth() lookups
qf.set_cache_offset_policy(Some(OffsetCacheKind::Linear { offset: 100 }));

// Add many elements
for i in 0usize..1000 {
    qf.add(&i.to_le_bytes()).unwrap();
}

// Jump to element 500 (much faster with caching)
let elem = qf.iter().nth(500);
assert!(elem.is_some());

drop(qf);
remove_file(path).ok();

Macros

ensure

Ensures that a condition is true, returning an error if it is not.

Structs

Iter

An iterator that yields the elements of a QueueFile from head to tail.

QueueFile

Enums

Error

The error type for [QueueFile] operations.

OffsetCacheKind

Policy controlling how element file-positions are cached to accelerate [Iter::nth].