Struct SChunk

pub struct SChunk(/* private fields */);

A super chunk (SChunk) is a collection of compressed chunks that are treated as a single entity.

It can be stored in memory or on disk, and support inserting, updating, deleting and appending chunks or data to it. It support random access to either chunks or items in the chunks.

Implementations

impl SChunk

pub fn new_in_memory(cparams: CParams, dparams: DParams) -> Result<Self, Error>

Create a new in-memory super chunk.

The created super chunk will not be contiguous. See Self::new for more details.

pub fn new_on_disk( urlpath: &Path, cparams: CParams, dparams: DParams, ) -> Result<Self, Error>

Create a new on-disk super chunk at the given path.

The created super chunk will not be contiguous. See Self::new for more details.

pub fn new( contiguous: bool, urlpath: Option<&Path>, cparams: CParams, dparams: DParams, ) -> Result<Self, Error>

Create a new super chunk with the specified parameters.

Arguments

• contiguous - If true, the super chunk will be stored in a contiguous memory block. Note that contiguous super chunks are very inefficient for update operations.
• urlpath - If Some(path), the super chunk will be stored on disk at the specified path. If None, the super chunk will be stored in memory.
• cparams - Compression parameters used to compress new chunks added to the super chunk.
• dparams - Decompression parameters used to decompress chunks from the super chunk.

pub fn open(urlpath: &Path) -> Result<Self, Error>

Open an existing super chunk from the specified path.

pub fn open_with_offset(urlpath: &Path, offset: usize) -> Result<Self, Error>

Open an existing super chunk from the specified path with an offset.

pub fn from_buffer(buffer: CowVec<'_, u8>) -> Result<Self, Error>

Create a super chunk from an existing in-memory buffer.

pub fn to_buffer(&mut self) -> Result<CowVec<'_, u8>, Error>

Serialize the super chunk to an in-memory buffer.

pub fn to_file(&mut self, urlpath: &Path, append: bool) -> Result<(), Error>

Serialize the super chunk to a file.

Arguments

• urlpath - The path to the file where the super chunk will be saved.
• append - If true, the super chunk will be appended to the file. If false, the file should not exist, otherwise an error will be returned.

pub fn append(&mut self, data: &[u8]) -> Result<(), Error>

Append (uncompressed) data to the super chunk as a new chunk.

The data will be compressed using the compression parameters of the super chunk.

pub fn append_chunk(&mut self, chunk: Chunk<'_>) -> Result<(), Error>

Append a compressed chunk to the super chunk.

pub fn update_chunk( &mut self, index: usize, chunk: Chunk<'_>, ) -> Result<(), Error>

Update an existing chunk in the super chunk.

The new chunk will replace the existing chunk at the specified index.

pub fn insert_chunk( &mut self, index: usize, chunk: Chunk<'_>, ) -> Result<(), Error>

Insert a new chunk at the specified index in the super chunk.

pub fn delete_chunk(&mut self, index: usize) -> Result<(), Error>

Delete a chunk at the specified index from the super chunk.

pub fn get_chunk(&mut self, index: usize) -> Result<Chunk<'_>, Error>

Get a chunk at the specified index from the super chunk.

pub fn decompress_chunk_into( &mut self, index: usize, dst: &mut [MaybeUninit<u8>], ) -> Result<usize, Error>

Decompress a chunk at the specified index into a new allocated bytes vector.

pub fn copy_to_memory( &self, cparams: CParams, dparams: DParams, ) -> Result<SChunk, Error>

Copy the super chunk to a new in-memory super chunk.

The created super chunk will not be contiguous. See Self::copy for more details.

pub fn copy_to_disk( &self, urlpath: &Path, cparams: CParams, dparams: DParams, ) -> Result<SChunk, Error>

Copy the super chunk to a new on-disk super chunk at the specified path.

The created super chunk will not be contiguous. See Self::copy for more details.

pub fn copy( &self, contiguous: bool, urlpath: Option<&Path>, cparams: CParams, dparams: DParams, ) -> Result<SChunk, Error>

Copy the super chunk to a new super chunk with the specified parameters.

Arguments

• contiguous - If true, the super chunk will be stored in a contiguous memory block. Note that contiguous super chunks are very inefficient for update operations.
• urlpath - If Some(path), the super chunk will be stored on disk at the specified path. If None, the super chunk will be stored in memory.
• cparams - Compression parameters used to compress new chunks added to the super chunk.
• dparams - Decompression parameters used to decompress chunks from the super chunk.

pub fn num_chunks(&self) -> usize

Get the number of chunks in the super chunk.

pub fn cparams(&self) -> CParams

Get the compression parameters used by this super chunk.

pub fn dparams(&self) -> DParams

Get the decompression parameters used by this super chunk.

pub fn typesize(&self) -> usize

Get the size of each item in the super chunk.

pub fn items_num(&self) -> usize

Get the number of items in the super chunk.

The returned number is the total number of items across all chunks in the super chunk. All of the item(s)(_into) and set_item(s) methods accept indices that are zero-based in the range 0..items_num().

pub fn item(&self, idx: usize) -> Result<Vec<u8>, Error>

Get an item at the specified index.

Each item is typesize (as provided during encoding) bytes long, and the index is zero-based.

Note that the returned vector may not be aligned to the original data type’s alignment, and the caller should ensure that the alignment is correct before transmuting it to original type. If the alignment does not match the original data type, the bytes should be copied to a new aligned allocation before transmuting, otherwise undefined behavior may occur. Alternatively, the caller can use Self::item_into and provide an already aligned destination buffer.

pub fn item_into( &self, idx: usize, dst: &mut [MaybeUninit<u8>], ) -> Result<usize, Error>

Get an item at the specified index and copy it into the provided destination buffer.

Each item is typesize (as provided during encoding) bytes long, and the index is zero-based.

Note that if the destination buffer is not aligned to the original data type’s alignment, the caller should not transmute the decompressed data to original type, as this may lead to undefined behavior.

pub fn items(&self, idx: Range<usize>) -> Result<Vec<u8>, Error>

Get a range of items specified by the index range.

Each item is typesize (as provided during encoding) bytes long, and the index is zero-based.

Note that the returned vector may not be aligned to the original data type’s alignment, and the caller should ensure that the alignment is correct before transmuting it to original type. If the alignment does not match the original data type, the bytes should be copied to a new aligned allocation before transmuting, otherwise undefined behavior may occur. Alternatively, the caller can use Self::items_into and provide an already aligned destination buffer.

pub fn items_into( &self, idx: Range<usize>, dst: &mut [MaybeUninit<u8>], ) -> Result<usize, Error>

Get a range of items specified by the index range and copy them into the provided destination buffer.

Each item is typesize (as provided during encoding) bytes long, and the index is zero-based.

Note that if the destination buffer is not aligned to the original data type’s alignment, the caller should not transmute the decompressed data to original type, as this may lead to undefined behavior.

pub fn set_item(&mut self, idx: usize, value: &[u8]) -> Result<(), Error>

Set an item at the specified index.

Note that this will re-compress the affected chunk(s) in the super chunk.

pub fn set_items( &mut self, idx: Range<usize>, values: &[u8], ) -> Result<(), Error>

Set a range of items specified by the index range.

Note that this will re-compress the affected chunk(s) in the super chunk.

Trait Implementations

impl Drop for SChunk

fn drop(&mut self)

Executes the destructor for this type.