Crate pco

source ·
Expand description

Pco (Pcodec) losslessly compresses and decompresses numerical sequences with high compression ratio and moderately fast speed.

§Quick Start

use pco::standalone::{simpler_compress, simple_decompress};
use pco::DEFAULT_COMPRESSION_LEVEL;
use pco::errors::PcoResult;

fn main() -> PcoResult<()> {
  // your data
  let mut my_nums = Vec::new();
  for i in 0..100000 {
    my_nums.push(i as i64);
  }

  // compress
  let compressed: Vec<u8> = simpler_compress(&my_nums, DEFAULT_COMPRESSION_LEVEL)?;
  println!("compressed down to {} bytes", compressed.len());

  // decompress
  let recovered = simple_decompress::<i64>(&compressed)?;
  println!("got back {} ints from {} to {}", recovered.len(), recovered[0], recovered.last().unwrap());
  Ok(())
}

§Compilation Notes

For best performance on x86_64, compile with any bmi* and avx* instruction sets your hardware supports. Almost all hardware x86 hardware these days supports bmi1, bmi2, and avx2. This improves compression speed slightly and decompression speed substantially! To make sure you’re using these, you can:

  • Add the following to your ~/.cargo/config.toml:
[target.'cfg(target_arch = "x86_64")']
rustflags = ["-C", "target-feature=+bmi1,+bmi2,+avx2"]
  • OR compile with RUSTFLAGS="-C target-feature=+bmi1,+bmi2,+avx2" cargo build --release ...

Note that settings target-cpu=native does not always have the same effect, since LLVM compiles for the lowest common denominator of instructions for a broad CPU family.

§API Notes

  • In some places, Pco methods accept a destination (either W: Write or &mut [T: NumberLike]). If Pco returns an error, it is possible both the destination and the struct have been modified.
  • Pco will always try to process all numbers, and it will fail if insufficient bytes are available. For instance, during decompression Pco will try to fill the entire &mut [T] passed in, returning an insufficient data error if the &[u8] passed in is not long enough.

Modules§

Macros§

  • A macro to help cross the dynamic<->generic boundary for pco core data types.
  • Similar to with_core_dtypes, but only for core latent types. Accepts a macro over a repeated list of $($name => $t,)+.

Structs§

Enums§

  • How Pco does the first step of processing for this chunk.
  • Specifies how Pco should choose a mode to compress this chunk of data.
  • PagingSpec specifies how a chunk is split into pages.

Constants§