Crate byteshuffle

Expand description

SIMD-accelerated byte shuffle/unshuffle routines

The byte-shuffle is a very efficient way to improve the compressibility of data that consists of an array of fixed-size objects. It rearranges the array in order to group all elements’ least significant bytes together, most-significant bytes together, and everything in between. Since real applications’ arrays often contain consecutive elements that are closely correlated with each other, this filter frequently results in lengthy continuous runs of identical bytes. Such runs are highly compressible by general-purpose compression libraries like gzip, lz4, etc.

The blosc project was the original inspiration for this library. Blosc is a C library intended primarily for HPC users, and it implements a shuffle filter, among many other things. This crate is a clean reimplementation of Blosc’s shuffle filter.

§Examples

Typical use: a byte array consists of an arithmetic sequence. Shuffle it, compress it, decompress it, and then unshuffle it.

const IN: [u8; 8] = [0x00, 0x01, 0x00, 0x02, 0x00, 0x03, 0x00, 0x04];
let shuffled = shuffle(2, &IN);
assert_ne!(IN, &shuffled[..]);
// In normal use, you would now serialize `shuffled`.  Then compress it, and later decompress
// and deserialize it.  Then unshuffle like below.
let unshuffled = unshuffle(2, &shuffled);
assert_eq!(IN, &unshuffled[..]);

§Crate Features

This crate has a “nightly” feature. It enables methods that require the use of types from the standard library that aren’t yet stabilized.

Structs§

ParseSimdImplErr

Enums§

SimdImpl: Explicitly specify an instruction set to use.

Functions§

select_implementation^⚠: Force the use of a particular CPU instruction set, globally.
shuffle: Shuffle a byte array whose contents are known to approximately repeat with period typesize.
shuffle_bufnightly: Like shuffle_into, but works with uninitialized output buffers.
shuffle_into: Like shuffle, but allows the caller to control allocation for the output.
shuffle_objects: Shuffle an array of fixed-size objects.
unshuffle: Unshuffle a byte array whose contents are known to approximately repeat with period typesize.
unshuffle_bufnightly: Like unshuffle_into, but works with uninitialized output buffers.
unshuffle_into: Like unshuffle, but allows the caller to control allocation for the destination.
unshuffle_objects^⚠: Unshuffle an array of fixed-size objects.