Crate shuffly 

Increases compressability of data with fixed-sized records.

Shuffly detects fixed-sized data patterns by trying out different pattern sized between (by default 1 to 64 bytes). For each pattern it reorders bytes such that byte X of each record is grouped together, and stores deltas of these bytes instead of the original data.

The resulting data stream is much more compressible for pattern based compression algorithms like deflate (gz, zip, etc), zstd, or lzma.

Shuffly is available both as a command line app (e.g. cargo install shuffly), and a library.

Library usage

let input: shuffly::Input = Box::new(File::open(input).unwrap());
let output: shuffly::Output = Box::new(File::create(output).unwrap());
let options = shuffly::Options::new();
if encode {
    shuffly::encode(0, input, output, &shuffly::Options::new()).unwrap();
} else {
    shuffly::decode(0, input, output).unwrap();
}

Structs

Options

Options passed to the encoder

Stats

Statistics about the encoding process

Enums

Error

Lists (non-exhaustivly) all possible error scenarios

Functions

decode

Decoded data that has previously been encoded. At least 3 threads will be used, one for input, one for output, and one for encoding

encode

Encodes data by searching for fixed-sized correlations in the data and shuffling data around accordingly. At least 3 threads will be used, one for input, one for output, and one for encoding

Type Aliases

Input

Output