buff-rs

A Rust implementation of BUFF: Decomposed Bounded Floats for Fast Compression and Queries.

Based on the VLDB 2021 paper: BUFF: Accelerating Queries in Memory through Decomposed Bounded Floats.

Overview

BUFF provides efficient compression and query execution for bounded floating-point data. Unlike general-purpose compression, BUFF is designed specifically for numeric data with known precision bounds (e.g., sensor readings, financial data), enabling:

1. Precision Bounding - Determines minimum bits needed to represent values within a given tolerance
2. Byte Slicing - Column-oriented storage where each bit position is stored contiguously
3. Direct Queries - Aggregate and filter operations execute directly on compressed data

Features

• Encode arrays of f64 values with configurable precision
• Decode back to f64 with controlled precision loss
• Encode arrays of decimal_bytes::Decimal values with configurable precision
• Decode back to decimal_bytes::Decimal with controlled precision loss
• Query compressed data directly (sum, max, count)
• Special value support: Infinity, -Infinity, NaN
• Columnar storage layout optimized for analytical queries
• Optional decimal-bytes interop for PostgreSQL NUMERIC compatibility
• Zero required dependencies (only thiserror)

Installation

Add to your Cargo.toml:

[dependencies]
buff-rs = "0.1"

Optional Features

[dependencies]
# Enable decimal-bytes interop for PostgreSQL NUMERIC compatibility
buff-rs = { version = "0.1", features = ["decimal"] }

Quick Start

use buff_rs::BuffCodec;

// Create a codec with 3 decimal places of precision (scale=1000)
let codec = BuffCodec::new(1000);

// Encode an array of f64 values
let data = vec![1.234, 5.678, 9.012, -3.456];
let encoded = codec.encode(&data).unwrap();

// Decode back to f64
let decoded = codec.decode(&encoded).unwrap();

// Query directly on compressed data
let sum = codec.sum(&encoded).unwrap();
let max = codec.max(&encoded).unwrap();

Choosing a Scale

The scale determines the precision of encoded values:

Choose a scale that matches your data's required precision. Higher scales provide more precision but may reduce compression ratio.

Special Values (Infinity, NaN)

BUFF supports special floating-point values:

use buff_rs::BuffCodec;

let codec = BuffCodec::new(1000);
let data = vec![1.0, f64::INFINITY, 2.0, f64::NAN, f64::NEG_INFINITY];

// Use encode_with_special for arrays containing special values
let encoded = codec.encode_with_special(&data).unwrap();
let decoded = codec.decode(&encoded).unwrap();

assert!(decoded[1].is_infinite());
assert!(decoded[3].is_nan());

Decimal-bytes Interop (PostgreSQL NUMERIC)

Enable the decimal feature for decimal-bytes compatibility:

[dependencies]
buff-rs = { version = "0.1", features = ["decimal"] }

use buff_rs::BuffCodec;
use decimal_bytes::Decimal;

let codec = BuffCodec::new(1000);

// Encode Decimal values (with precision loss)
let decimals: Vec<Decimal> = vec![
    "1.234".parse().unwrap(),
    "5.678".parse().unwrap(),
];
let encoded = codec.encode_decimals(&decimals).unwrap();

// Decode back to Decimal
let decoded: Vec<Decimal> = codec.decode_to_decimals(&encoded).unwrap();

Warning: Converting between BUFF and Decimal involves precision loss because BUFF uses bounded floating-point representation while Decimal uses exact arbitrary-precision.

Performance

Key performance characteristics (run cargo bench locally for your hardware):

Compression Ratio

(For 10,000 f64 values = 80 KB uncompressed)

Comparison with decimal-bytes

For 1,000 values with 3 decimal places:

BUFF provides ~2.5x better compression and ~96x faster array decoding for columnar workloads, while decimal-bytes is optimized for individual value operations with lexicographic ordering.

When to Use BUFF vs decimal-bytes

Use decimal-bytes when:

• You need exact arbitrary-precision decimals (like PostgreSQL NUMERIC)
• Values are stored individually in documents
• You need lexicographically sortable byte representation

Use buff-rs when:

• You have arrays of floating-point values with known precision
• You're building columnar storage or time-series databases
• You want to query compressed data without full decompression

How It Works

Precision Bounding

Given a precision tolerance (e.g., 0.001 for 3 decimal places), BUFF determines the minimum number of bits needed to represent each value. This is done by analyzing the IEEE 754 representation and finding the position where further bits don't affect precision.

Byte Slicing

Instead of storing values row-by-row, BUFF stores them column-by-column at the byte level:

Traditional:  [v1_byte0, v1_byte1] [v2_byte0, v2_byte1] [v3_byte0, v3_byte1]
Byte-sliced:  [v1_byte0, v2_byte0, v3_byte0] [v1_byte1, v2_byte1, v3_byte1]

This layout enables SIMD-accelerated comparisons for range queries.

Sign Flipping

To enable proper ordering of negative and positive values, the sign bit is flipped during encoding. This ensures that comparing encoded bytes yields correct numerical ordering.

Compression Performance

Compression ratio depends on data characteristics:

• Narrow value range: Better compression (fewer bits needed for delta)
• Wide value range: More bits needed, lower compression
• Repetitive values: Good compression (single base value)

Typical compression ratios range from 0.3x to 0.8x of the original size (8 bytes per f64).

License

MIT License - see LICENSE for details.

References

• Chunwei Liu, et al. "Decomposed Bounded Floats for Fast Compression and Queries." VLDB 2021. Paper
• Original implementation: Tranway1/buff