Crate pco

Source
Expand description
Pco logo: a pico-scale, compressed version of the Pyramid of Khafre in the palm of your hand

Pco (Pcodec) losslessly compresses and decompresses numerical sequences with high compression ratio and moderately fast speed.

§Quick Start

use pco::standalone::{simpler_compress, simple_decompress};
use pco::DEFAULT_COMPRESSION_LEVEL;
use pco::errors::PcoResult;

fn main() -> PcoResult<()> {
  // your data
  let mut my_nums = Vec::new();
  for i in 0..100000 {
    my_nums.push(i as i64);
  }

  // compress
  let compressed: Vec<u8> = simpler_compress(&my_nums, DEFAULT_COMPRESSION_LEVEL)?;
  println!("compressed down to {} bytes", compressed.len());

  // decompress
  let recovered = simple_decompress::<i64>(&compressed)?;
  println!("got back {} ints from {} to {}", recovered.len(), recovered[0], recovered.last().unwrap());
  Ok(())
}

§Compilation Notes

For best performance on x86_64, compile with any bmi* and avx* instruction sets your hardware supports. Almost all x86_64 hardware these days supports bmi1, bmi2, and avx2. This improves compression speed slightly and decompression speed substantially! To make sure you’re using these, you can:

  • Add the following to your ~/.cargo/config.toml:
[target.'cfg(target_arch = "x86_64")']
rustflags = ["-C", "target-feature=+bmi1,+bmi2,+avx2"]
  • OR compile with RUSTFLAGS="-C target-feature=+bmi1,+bmi2,+avx2" cargo build --release ...

Note that settings target-cpu=native does not always have the same effect, since LLVM compiles for the lowest common denominator of instructions for a broad CPU family.

§API Notes

  • In some places, Pco methods accept a destination (either W: Write or &mut [T: Number]). If Pco returns an error, it is possible both the destination and the struct have been modified.
  • Pco will always try to process all numbers, and it will fail if insufficient bytes are available. For instance, during decompression Pco will try to fill the entire &mut [T] passed in, returning an insufficient data error if the &[u8] passed in is not long enough.

Modules§

  • data_types
  • describers
    for inspecting certain types of Pco metadata
  • errors
  • metadata
    structs representing stored information about how compression was done
  • standalone
    for compressing/decompressing .pco files
  • wrapped
    for compressing/decompressing as part of an outer, wrapping format

Macros§

  • define_number_enum
    unstable API Defines enums holding a container generic to Number.
  • match_latent_enum
    Matches enums holding a container of Latents and puts the concrete type into scope.
  • match_number_enum
    Matches enums holding a container of Numbers and puts the concrete type into scope.

Structs§

  • ChunkConfig
    All configurations available for a compressor.
  • Progress
    Information about progress after calling a decompression function.

Enums§

  • DeltaSpec
    Specifies how Pco should choose a delta encoding to compress this chunk of data.
  • ModeSpec
    Specifies how Pco should choose a mode to compress this chunk of data.
  • PagingSpec
    PagingSpec specifies how a chunk is split into pages.

Constants§