Crate pco

Pco (Pcodec) losslessly compresses and decompresses numerical sequences with high compression ratio and moderately fast speed.

Quick Start

use pco::standalone::{simpler_compress, simple_decompress};
use pco::DEFAULT_COMPRESSION_LEVEL;
use pco::errors::PcoResult;

fn main() -> PcoResult<()> {
  // your data
  let mut my_nums = Vec::new();
  for i in 0..100000 {
    my_nums.push(i as i64);
  }

  // compress
  let compressed: Vec<u8> = simpler_compress(&my_nums, DEFAULT_COMPRESSION_LEVEL)?;
  println!("compressed down to {} bytes", compressed.len());

  // decompress
  let recovered: Vec<i64> = simple_decompress(&compressed)?;
  assert_eq!(recovered, my_nums);
  Ok(())
}

Compilation Notes

**For best performance on x86_64, compile with bmi1, bmi2, and avx2. This improves compression speed slightly and decompression speed substantially! Almost all hardware nowadays supports these instruction sets. To make sure you’re using these, you can:

• Add the following to your ~/.cargo/config.toml:

[target.'cfg(target_arch = "x86_64")']
rustflags = ["-C", "target-feature=+bmi1,+bmi2,+avx2"]

• OR compile with RUSTFLAGS="-C target-feature=+bmi1,+bmi2,+avx2" cargo build --release ...

Note that setting target-cpu=native does not always have the same effect, since LLVM compiles for the lowest common denominator of instructions for a broad CPU family.

API Notes

• In some places, Pco methods accept a destination (either W: Write or &mut [T: Number]). If Pco returns an error, it is possible both the destination and the struct have been modified.
• Pco will always try to process all numbers, and it will fail if insufficient bytes are available. For instance, during decompression Pco will try to fill the entire &mut [T] passed in, returning an insufficient data error if the &[u8] passed in is not long enough.

Modules

data_types

describers

for inspecting certain types of Pco metadata

errors

metadata

structs representing stored information about how compression was done

standalone

for compressing/decompressing .pco files

wrapped

for compressing/decompressing as part of an outer, wrapping format

Macros

define_number_enum

unstable API Defines enums holding a container generic to Number.

match_latent_enum

Matches enums holding a container of Latents and puts the concrete type into scope.

match_number_enum

Matches enums holding a container of Numbers and puts the concrete type into scope.

Structs

ChunkConfig

All configurations available for a compressor.

Progress

Information about progress after calling a decompression function.

Enums

DeltaSpec

Specifies how Pco should choose a delta encoding to compress this chunk of data.

ModeSpec

Specifies how Pco should choose a mode to compress this chunk of data.

PagingSpec

PagingSpec specifies how a chunk is split into pages.

Constants

DEFAULT_COMPRESSION_LEVEL

DEFAULT_MAX_PAGE_N

FULL_BATCH_N

The count of numbers per batch, the smallest unit of decompression.