Struct zuffer::Zuffer

pub struct Zuffer { /* private fields */ }

Zuffer is equivalent of Go’s bytes.Zuffer without the ability to read. It is NOT thread-safe.

In Memory mode, default allocator is used to allocate memory, which depending upon how the code is compiled could use jemalloc for allocations.

In Mmap mode, Zuffer uses file mmap to allocate memory. This allows us to store big data structures without using physical memory.

max_size can be set to limit the memory usage.

Implementations

impl Zuffer

pub const DEFAULT_CAPACITY: usize = 64usize

default capacity of the buffer

pub fn new(capacity: usize, tag: &'static str) -> Self

Returns a memory buffer with default capacity

NOTE:

It is the caller’s responsibility to set offset after this, because Zuffer doesn’t remember what it was.

pub fn persistent<P: AsRef<Path>>(
    capacity: usize,
    file: P
) -> Result<Self, Error>

Returns a persistent buffer with default capacity

NOTE:

It is the caller’s responsibility to set offset after this, because Zuffer doesn’t remember what it was.

pub fn temp<P: AsRef<Path>>(
    capacity: usize,
    dir: Option<P>
) -> Result<Self, Error>

Returns a temperary buffer with default capacity, the underlying temp file will be removed when the struct is dropped.

NOTE:

It is the caller’s responsibility to set offset after this, because Zuffer doesn’t remember what it was.

pub fn with_auto_mmap<NP>(self, threshold: usize, path: NP) -> Selfwhere
    NP: AsRef<Path>,

Returns a memory buffer with default capacity, but after the size exceeds the threshold, then the buffer will auto mmap to a temp file.

NOTE:

It is the caller’s responsibility to set offset after this, because Zuffer doesn’t remember what it was.

pub fn with_max_size(self, size: usize) -> Self

Set the maximum size for the buffer.

pub fn is_empty(&self) -> bool

Returns if the buffer is empty

pub fn start_offset(&self) -> i64

Returns the buffer start offset

pub fn tag(&self) -> &'static str

Returns the tag of the buffer

pub fn set_tag(&mut self, tag: &'static str)

Set the tag for the buffer

pub fn len_with_padding(&self) -> usize

Returns the number of bytes written to the buffer so far plus the padding at the start of the buffer.

pub fn len_without_padding(&self) -> usize

Returns the number of bytes written to the buffer so far (without the padding).

pub fn bytes(&self) -> &[u8]

Returns all the written bytes as a slice

pub fn bytes_mut(&mut self) -> &mut [u8]

Returns all the written bytes as a mutable slice

pub fn data(&self, offset: usize) -> Result<&[u8], Error>

Returns the underlying data in range [offset..current_size - offset]

pub fn data_mut(&mut self, offset: usize) -> Result<&mut [u8], Error>

Returns the mutable underlying data in range [offset..current_size - offset]

pub fn grow(&mut self, n: usize) -> Result<(), Error>

Grow would grow the buffer to have at least n more bytes. In case the buffer is at capacity, it would reallocate twice the size of current capacity + n, to ensure n bytes can be written to the buffer without further allocation. In UseMmap mode, this might result in underlying file expansion.

pub fn allocate(&mut self, n: usize) -> Result<&mut [u8], Error>

allocate is a way to get a slice of size n back from the buffer. This slice can be directly written to.

Warning:

Allocate is not thread-safe. The byte slice returned MUST be used before further calls to Zuffer.

pub fn allocate_offset(&mut self, n: usize) -> Result<usize, Error>

allocate_offset works the same way as allocate, but instead of returning a byte slice, it returns the offset of the allocation.

pub fn slice_allocate(&mut self, size: usize) -> Result<&mut [u8], Error>

slice_allocate would encode the size provided into the buffer, followed by a call to allocate, hence returning the slice of size sz. This can be used to allocate a lot of small buffers into this big buffer. Note that slice_allocate should NOT be mixed with normal calls to write.

pub fn write(&mut self, p: &[u8]) -> Result<usize, Error>

write would write p bytes to the buffer.

pub fn write_bytes(&mut self, bytes: &[u8]) -> Result<(), Error>

Write all bytes to the buffer

pub fn reset(&self)

reset would reset the buffer to be reused.

pub fn slice(&self, offset: i64) -> (&[u8], i64)

slice would return the slice written at offset.

pub fn slice_mut(&mut self, offset: i64) -> (&mut [u8], i64)

slice_mut would return the slice written at offset.

pub fn slice_offsets(&self) -> Vec<i64>

slice_offsets is an expensive function. Use sparingly.

pub fn slice_iterate<F>(&self, f: F) -> Result<(), Error>where
    F: FnMut(&[u8]) -> Result<(), Error>,

Iterates over the buffer

pub fn slice_iterate_mut<F>(&mut self, f: F) -> Result<(), Error>where
    F: Fn(&mut [u8]) -> Result<(), Error>,

Iterates over the buffer

pub fn sort_slice<L>(&mut self, less: L) -> Result<(), Error>where
    L: Fn(&[u8], &[u8]) -> bool,

Sort the buffer

pub fn sort_slice_between<L>(
    &mut self,
    start: i64,
    end: i64,
    less: L
) -> Result<(), Error>where
    L: Fn(&[u8], &[u8]) -> bool,

Sort the buffer between the given offsets

Trait Implementations

impl Drop for Zuffer

fn drop(&mut self)

Executes the destructor for this type.