Crate vortex_buffer

Expand description

A library for working with custom aligned buffers of sized values.

The vortex-buffer crate supports zero-copy cloning and slicing with a custom allocator and runtime alignment.

Buffer<T> and BufferMut<T> provide immutable and mutable typed buffers.
ByteBuffer and ByteBufferMut are type aliases for u8 buffers.
BufferString is a wrapper around a ByteBuffer that enforces utf-8 encoding.
ConstBuffer<T, const A: usize> provides similar functionality to Buffer<T> except with a compile-time alignment of A.
buffer! and buffer_mut! macros with the same syntax as the builtin vec! macro for inline construction of buffers.
BitBuffer and BitBufferMut provide packed bitsets that can be used to store boolean values.

You can think of BufferMut<T> as similar to a Vec<T>, except that any operation that may cause a re-allocation, e.g. extend, will ensure the new allocation maintains the buffer’s defined alignment.

For example, it’s possible to incrementally build a Buffer<T> with a 4KB alignment.

use vortex_buffer::{Alignment, BufferMut};

let mut buf = BufferMut::<i32>::empty_aligned(Alignment::new(4096));
buf.extend(0i32..1_000);
assert_eq!(buf.as_ptr().align_offset(4096), 0)

§Comparison

Implementation	Zero-copy	Custom Alignment	Typed
`vortex_buffer::Buffer<T>`	✔️	✔️	✔️
`arrow_buffer::ScalarBuffer<T>`	✔️	❌️️️	✔️
`bytes::Bytes`	✔️	❌️️️	❌️️️
`Vec<T>`	❌️	❌️️	✔️

§Features

The arrow feature can be enabled to provide conversion functions to/from Arrow Rust buffers, including arrow_buffer::Buffer, arrow_buffer::ScalarBuffer<T>, and arrow_buffer::OffsetBuffer.

Modules§

trusted_len: Trusted-length iterator trait and adapters for safe pre-allocation.

Macros§

bitbuffer: A macro for constructing bit-buffers akin to vec![..].
bitbuffer_mut: A macro for constructing bit-buffers akin to vec![..].
buffer: A macro for constructing buffers akin to vec![..].
buffer_mut: A macro for constructing buffers akin to vec![..].

Structs§

Alignment: The alignment of a buffer.
BitBuffer: An immutable bitset stored as a packed byte buffer.
BitBufferMeta: In-memory metadata describing a packed bitset: a normalized bit offset (always < 8) and a logical bit len.
BitBufferMut: A mutable bitset buffer that allows random access to individual bits for set and get.
BitBufferMutView: A mutable, borrowed view over a packed bitset.
BitBufferView: An immutable, borrowed view over a packed bitset.
BitChunkIterator: Iterator over chunks of 64 bits represented as an u64
BitChunks: Iterates over an arbitrarily aligned byte buffer 64 bits at a time
BitIndexIterator: An iterator of usize whose index in a provided bitmask is true
BitIterator: Iterator over the bits within a packed bitmask
BitSliceIterator: Iterator of contiguous ranges of set bits within a provided packed bitmask
Buffer: An immutable buffer of items of T.
BufferAllocatorRef: A shared reference to a buffer allocator.
BufferIterator: Owned iterator over a Buffer.
BufferMut: A mutable buffer that maintains a runtime-defined alignment through resizing operations.
BufferString: A wrapper around a ByteBuffer that guarantees that the buffer contains valid UTF-8.
ConstBuffer: A buffer of items of T with a compile-time alignment.
CpuKernel: A function pointer selected by CPU-feature detection once, on first use.
Iter: An iterator over Buffer elements.
StaticBufferAllocator: The allocator used by buffer APIs that do not take an allocator.
UnalignedBitChunk: Iterates over an arbitrarily aligned byte buffer

Traits§

AlignedBuf: An extension to the Buf trait that provides a function copy_to_aligned similar to copy_to_bytes that allows for zero-copy aligned reads where possible.
BufferAllocator: An allocator that can back a Vortex buffer.

Functions§

collect_bool_word: Packs up to 64 boolean values into a little-endian u64 word.
collect_bool_word_scalar: Packs up to 64 boolean values into a little-endian u64 word one bit at a time.
collect_bool_words: Pack len boolean values returned by f into the prefix of words, LSB-first, 64 bits per u64. words must have capacity for at least len.div_ceil(64) entries.
collect_bool_words_avx2^⚠: AVX2 copy of the collect_bool_words word loop.
collect_bool_words_avx512^⚠: AVX-512BW copy of the collect_bool_words word loop.
collect_bool_words_multiversioned: Word loop with the widest pack kernel the CPU offers (AVX-512BW, then AVX2, then the baseline), for predicates known to be cheap.
collect_bool_words_sse2^⚠: SSE2 copy of the collect_bool_words word loop.
get_bit: Get the bit value at index out of buf.
get_bit_unchecked^⚠: Get the bit value at index out of buf without bounds checking.
pack_bool_word_avx2^⚠: AVX2 byte→bit pack kernel: two 32-byte vpcmpeqb-against-zero + vpmovmskb rounds.
pack_bool_word_avx512^⚠: AVX-512BW byte→bit pack kernel: a single 64-byte vptestmb produces the whole word.
pack_bool_word_sse2^⚠: SSE2 byte→bit pack kernel: four 16-byte pcmpeqb-against-zero + pmovmskb rounds.
pack_bool_word_swar: Portable branch-free byte→bit pack kernel, used when no SIMD kernel is available.
pack_bools_into_words: Pack len boolean values returned by f into words starting at bit position bit_offset, LSB-first.
read_u64_le: Read up to 8 bytes as a little-endian u64, zero-padding the high bytes when fewer than 8 bytes are supplied.
set_bit_unchecked^⚠: Set the bit value at index in buf without bounds checking.
splice_word_at_bit: Splice a packed word w (whose bits above the highest valid bit are zero) into words at the given bit position.
unset_bit_unchecked^⚠: Unset the bit value at index in buf without bounds checking.

Type Aliases§

ByteBuffer: An immutable buffer of u8.
ByteBufferMut: A mutable buffer of u8.
ConstByteBuffer: A const-aligned buffer of u8.
UnalignedBitChunkIterator: Iterator over an UnalignedBitChunk