Struct QueueFile

pub struct QueueFile { /* private fields */ }

QueueFile is a lightning-fast, transactional, file-based FIFO.

Addition and removal from an instance is an O(1) operation and is atomic. Writes are synchronous by default; data will be written to disk before an operation returns.

The underlying file. Uses a ring buffer to store entries. Designed so that a modification isn’t committed or visible until we write the header. The header is much smaller than a segment. So long as the underlying file system supports atomic segment writes, changes to the queue are atomic. Storing the file length ensures we can recover from a failed expansion (i.e. if setting the file length succeeds but the process dies before the data can be copied).

Example

use queue_file::QueueFile;

let mut qf = QueueFile::open(path)
    .expect("cannot open queue file");
let data = "Welcome to QueueFile!".as_bytes();

qf.add(&data).expect("add failed");

if let Ok(Some(bytes)) = qf.peek() {
    assert_eq!(data, bytes.as_ref());
}

qf.remove().expect("remove failed");

File format

  16-32 bytes      Header
  ...              Data

This implementation supports two versions of the header format.

Versioned Header (32 bytes):
  1 bit            Versioned indicator [0 = legacy, 1 = versioned]
  31 bits          Version, always 1
  8 bytes          File length
  4 bytes          Element count
  8 bytes          Head element position
  8 bytes          Tail element position

Legacy Header (16 bytes):
  1 bit            Legacy indicator, always 0
  31 bits          File length
  4 bytes          Element count
  4 bytes          Head element position
  4 bytes          Tail element position

Each element stored is represented by:

Element:
  4 bytes          Data length
  ...              Data

Implementations

impl QueueFile

pub fn with_capacity<P: AsRef<Path>>( path: P, capacity: u64, ) -> Result<Self, Error>

Open or create QueueFile at path with specified minimal file size.

Example

let qf = QueueFile::with_capacity(path, 120).expect("failed to open queue");

pub fn open<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Open or create QueueFile at path.

Example

let qf = QueueFile::open(path).expect("failed to open queue");

pub fn open_legacy<P: AsRef<Path>>(path: P) -> Result<Self, Error>

Open or create QueueFile at path forcing legacy format.

Example

let qf = QueueFile::open_legacy(path).expect("failed to open queue");

pub const fn overwrite_on_remove(&self) -> bool

Returns true if removing an element will also overwrite data with zero bytes.

pub const fn get_overwrite_on_remove(&self) -> bool

👎Deprecated since 1.4.7: Use overwrite_on_remove instead.

pub fn set_overwrite_on_remove(&mut self, value: bool)

If set to true removing an element will also overwrite data with zero bytes.

pub const fn sync_writes(&self) -> bool

Returns true if every write to file will be followed by sync_data() call.

pub const fn get_sync_writes(&self) -> bool

👎Deprecated since 1.4.7: Use sync_writes instead.

pub fn set_sync_writes(&mut self, value: bool)

If set to true every write to file will be followed by sync_data() call.

pub const fn skip_write_header_on_add(&self) -> bool

Returns true if skips header update upon adding enabled.

pub const fn get_skip_write_header_on_add(&self) -> bool

👎Deprecated since 1.4.7: Use skip_write_header_on_add instead.

pub fn set_skip_write_header_on_add(&mut self, value: bool)

If set to true skips header update upon adding.

pub fn set_read_buffer_size(&mut self, size: usize)

Changes buffer size used for data reading.

pub const fn cache_offset_policy(&self) -> Option<OffsetCacheKind>

pub const fn get_cache_offset_policy(&self) -> Option<OffsetCacheKind>

👎Deprecated since 1.4.7: Use cache_offset_policy instead.

pub fn set_cache_offset_policy( &mut self, kind: impl Into<Option<OffsetCacheKind>>, )

pub const fn is_empty(&self) -> bool

Returns true if this queue contains no entries.

pub const fn size(&self) -> usize

Returns the number of elements in this queue.

pub fn sync_all(&mut self) -> Result<(), Error>

Synchronizes the underlying file, look at File::sync_all doc for more info.

pub fn add_n( &mut self, elems: impl IntoIterator<Item = impl AsRef<[u8]>> + Clone, ) -> Result<(), Error>

pub fn add(&mut self, buf: &[u8]) -> Result<(), Error>

Adds an element to the end of the queue.

pub fn peek(&mut self) -> Result<Option<Box<[u8]>>, Error>

Reads the eldest element. Returns OK(None) if the queue is empty.

pub fn remove(&mut self) -> Result<(), Error>

Removes the eldest element.

pub fn remove_n(&mut self, n: usize) -> Result<(), Error>

Removes the eldest n elements.

pub fn clear(&mut self) -> Result<(), Error>

Clears this queue. Truncates the file to the initial size.

pub fn iter(&mut self) -> Iter<'_>

Returns an iterator over elements in this queue.

Example

let mut qf = QueueFile::open(path).expect("failed to open queue");
let items = vec![vec![1, 2], vec![], vec![3]];
qf.add_n(&items).expect("failed to add elements to queue");

let stored = qf.iter().map(Vec::from).collect::<Vec<_>>();
assert_eq!(items, stored);

pub const fn file_len(&self) -> u64

Returns the amount of bytes used by the backed file. Always >= Self::used_bytes.

pub const fn used_bytes(&self) -> u64

Returns the amount of bytes used by the queue.

pub fn into_inner_file(self) -> File

Returns underlying file of the queue.

Trait Implementations

impl Debug for QueueFile

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for QueueFile

fn drop(&mut self)

Executes the destructor for this type.