Crate vortex_buffer

Expand description

A library for working with custom aligned buffers of sized values.

The vortex-buffer crate is built around bytes::Bytes and therefore supports zero-copy cloning and slicing, but differs in that it can define and maintain a custom alignment.

Buffer<T> and BufferMut<T> provide immutable and mutable wrappers around bytes::Bytes and bytes::BytesMut respectively.
ByteBuffer and ByteBufferMut are type aliases for u8 buffers.
BufferString is a wrapper around a ByteBuffer that enforces utf-8 encoding.
ConstBuffer<T, const A: usize> provides similar functionality to Buffer<T> except with a compile-time alignment of A.
buffer! and buffer_mut! macros with the same syntax as the builtin vec! macro for inline construction of buffers.
BitBuffer and BitBufferMut provide packed bitsets that can be used to store boolean values.

You can think of BufferMut<T> as similar to a Vec<T>, except that any operation that may cause a re-allocation, e.g. extend, will ensure the new allocation maintains the buffer’s defined alignment.

For example, it’s possible to incrementally build a Buffer<T> with a 4KB alignment.

use vortex_buffer::{Alignment, BufferMut};

let mut buf = BufferMut::<i32>::empty_aligned(Alignment::new(4096));
buf.extend(0i32..1_000);
assert_eq!(buf.as_ptr().align_offset(4096), 0)

§Comparison

Implementation	Zero-copy	Custom Alignment	Typed
`vortex_buffer::Buffer<T>`	✔️	✔️	✔️
`arrow_buffer::ScalarBuffer<T>`	✔️	❌️️️	✔️
`bytes::Bytes`	✔️	❌️️️	❌️️️
`Vec<T>`	❌️	❌️️	✔️

§Features

The arrow feature can be enabled to provide conversion functions to/from Arrow Rust buffers, including arrow_buffer::Buffer, arrow_buffer::ScalarBuffer<T>, and arrow_buffer::OffsetBuffer.

Modules§

trusted_len: Trusted-length iterator trait and adapters for safe pre-allocation.

Macros§

bitbuffer: A macro for constructing bit-buffers akin to vec![..].
bitbuffer_mut: A macro for constructing bit-buffers akin to vec![..].
buffer: A macro for constructing buffers akin to vec![..].
buffer_mut: A macro for constructing buffers akin to vec![..].

Structs§

Alignment: The alignment of a buffer.
BitBuffer: An immutable bitset stored as a packed byte buffer.
BitBufferMeta: In-memory metadata describing a packed bitset: a normalized bit offset (always < 8) and a logical bit len.
BitBufferMut: A mutable bitset buffer that allows random access to individual bits for set and get.
BitBufferMutView: A mutable, borrowed view over a packed bitset.
BitBufferView: An immutable, borrowed view over a packed bitset.
BitChunkIterator: Iterator over chunks of 64 bits represented as an u64
BitChunks: Iterates over an arbitrarily aligned byte buffer 64 bits at a time
BitIndexIterator: An iterator of usize whose index in a provided bitmask is true
BitIterator: Iterator over the bits within a packed bitmask
BitSliceIterator: Iterator of contiguous ranges of set bits within a provided packed bitmask
Buffer: An immutable buffer of items of T.
BufferIterator: Owned iterator over a Buffer.
BufferMut: A mutable buffer that maintains a runtime-defined alignment through resizing operations.
BufferString: A wrapper around a ByteBuffer that guarantees that the buffer contains valid UTF-8.
ConstBuffer: A buffer of items of T with a compile-time alignment.
Iter: An iterator over Buffer elements.
UnalignedBitChunk: Iterates over an arbitrarily aligned byte buffer

Constants§

ALIGNMENT_TO_HOST_COPY: Default alignment for device-to-host buffer copies.

Traits§

AlignedBuf: An extension to the Buf trait that provides a function copy_to_aligned similar to copy_to_bytes that allows for zero-copy aligned reads where possible.

Functions§

collect_bool_word: Packs up to 64 boolean values into a little-endian u64 word.
collect_bool_words: Pack len boolean values returned by f into the prefix of words, LSB-first, 64 bits per u64. words must have capacity for at least len.div_ceil(64) entries.
get_bit: Get the bit value at index out of buf.
get_bit_unchecked^⚠: Get the bit value at index out of buf without bounds checking.
pack_bools_into_words: Pack len boolean values returned by f into words starting at bit position bit_offset, LSB-first.
set_bit_unchecked^⚠: Set the bit value at index in buf without bounds checking.
splice_word_at_bit: Splice a packed word w (whose bits above the highest valid bit are zero) into words at the given bit position.
unset_bit_unchecked^⚠: Unset the bit value at index in buf without bounds checking.

Type Aliases§

ByteBuffer: An immutable buffer of u8.
ByteBufferMut: A mutable buffer of u8.
ConstByteBuffer: A const-aligned buffer of u8.
UnalignedBitChunkIterator: Iterator over an UnalignedBitChunk