fastpfor 0.6.3

FastPFOR library written in Rust.
Documentation

FastPFor for Rust

GitHub crates.io version docs.rs docs license CI build

This is a Rust wrapper for the C++ FastPFor library, as well as a pure Rust re-implementation (work in progress). Supports 32-bit and 64-bit integers, and SIMD-optimized codecs for 128-bit and 256-bit vectors. Based on the Decoding billions of integers per second through vectorization, 2012 paper.

Supported algorithms

Unless otherwise specified, all codecs support &[u32] only.

* BP32
* Copy
* FastBinaryPacking8
* FastPFor128 (both `&[u32]` and `&[u64]`)
* FastPFor256 (both `&[u32]` and `&[u64]`)
* FastBinaryPacking16
* FastBinaryPacking32
* MaskedVByte
* NewPFor
* OptPFor
* PFor2008
* PFor
* SimdBinaryPacking
* SimdFastPFor128
* SimdFastPFor256
* SimdGroupSimple
* SimdGroupSimpleRingBuf
* SimdNewPFor
* SimdOptPFor
* SimdPFor
* SimdSimplePFor
* Simple16
* Simple8b
* Simple8bRle
* Simple9
* Simple9Rle
* SimplePFor
* StreamVByte
* VByte
* VarInt (both `&[u32]` and `&[u64]`)
* VarIntGb

Usage

Crate Features

  • cpp - C++ implementation (default)
  • rust - Rust implementation (work in progress, opt-in)

Using C++ Wrapper

use fastpfor::cpp::{Codec32 as _, SimdFastPFor128Codec};

fn main() {
  let mut codec = SimdFastPFor128Codec::new();

  // Encode
  let mut input = vec![1, 2, 3, 4, 5];
  let mut output = vec![0; 10];  // must be large enough
  let enc_slice = codec.encode32(&input, &mut output).unwrap();

  // Decode
  let mut decoded = vec![0; 10]; // must be large enough
  let dec_slice = codec.decode32(&enc_slice, &mut decoded).unwrap();

  assert_eq!(input, dec_slice);
}

Build Requirements

When using the Rust implementation, no additional dependencies are required. When using the C++ implementation, you need to have a C++ compiler that supports C++14 and SIMD intrinsics. See FastPFor C++ requirements.

Linux

The default GitHub action runner for Linux has all the needed dependencies. For local development, you may need to install the following packages.

# This list may be incomplete
sudo apt-get install build-essential libsimde-dev

macOS

To build FastPFor on macOS, you'll need to install SIMDe. Since Homebrew installs packages in /opt/homebrew (for Apple Silicon), you'll also need to explicitly set the include paths.

# install SIMDe via Homebrew
brew install simde

# Ensure the compiler can find the required headers before building
export CXXFLAGS="-I/opt/homebrew/include"
export CFLAGS="-I/opt/homebrew/include"

Development

  • This project is easier to develop with just, a modern alternative to make. Install it with cargo install just.
  • To get a list of available commands, run just.
  • To run tests, use just test.

License

Licensed under either of

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual-licensed as above, without any additional terms or conditions.