fastalp 0.1.25

High-performance lossless floating-point compression in pure Rust / 基于 ALP 算法的高性能无损浮点数压缩库
Documentation

English | 中文

crates.io docs.rs


fastalp : Adaptive Lossless Floating-Point Compression in Rust

Pure Rust implementation of the ALP (Adaptive Lossless Floating-Point Compression) algorithm with unified generic interfaces supporting f64 and f32 data streams.


Overview

Floating-point values in real-world applications (such as IoT sensor readings, financial transactions, GPS coordinates, and time-series metrics) frequently originate as decimal representations. Traditional general-purpose compression algorithms and integer bitpackers operate inefficiently on IEEE 754 representations due to distributed exponent and mantissa bit patterns.

fastalp implements the ALP compression algorithm:

  • Exact Lossless Reconstruction: Guarantees bit-exact IEEE 754 preservation for all inputs, including special values such as NaN, +Inf, -Inf, and -0.0.

  • Adaptive Delta Differential Encoding (Delta-ALP): Automatically evaluates smooth and continuous physical time series (weather, hydrology, telemetry), adaptively applying first-order differences and branchless prefix sum accumulation to reduce bit widths by 15% to 38%.

  • Decimal Division Exact Mode: Completely eliminates IEEE 754 multiplication roundoff errors (such as * 0.1) by reconstructing via exact decimal division, driving outlier exception counts to zero on real-world telemetry.

  • Stack-Allocated LUT & SIMD Hybrid Decompression: Utilizes 256-entry stack lookup tables for division modes to eliminate hardware division latencies, coupled with pure-register SIMD auto-vectorization for linear arithmetic exceeding 55+ GB/s throughput.

  • Adaptive Parameter Estimation: Samples input sequences to derive optimal scaling parameters (exp, fac, use_div) that minimize bit-width requirements.

  • Frame-of-Reference & Bitpacking: Encodes converted integers using base subtraction (FOR / Delta) and dense bit-packing from 1 to 64 bits per value.

  • Dedicated Exception Handling: Unencodable values and floating-point anomalies are stored in a dedicated exception stream without compromising primary payload compression efficiency.

  • Raw Fallback Protection: Automatically falls back to uncompressed raw mode when noise or extreme precision values would cause negative compression.

  • Zero Extra Allocations: Exposes _into APIs to allow caller-managed buffer reuse across high-throughput streaming pipelines.

  • Unified Generic Interface: compress, compress_into, decompress, and decompress_into work across both f64 and f32.


Usage

Installation

cargo add fastalp

Basic Compression and Decompression

use fastalp::{compress, decompress, Result};

fn main() -> Result<()> {
  let sensor_data = vec![20.5, 20.6, 20.8, 21.0, 20.9, 21.2];

  // Compress floating-point slice into byte buffer (generic for f64 / f32)
  let compressed = compress(&sensor_data);

  // Decompress byte buffer back to exact f64 slice
  let decompressed: Vec<f64> = decompress(&compressed)?;

  assert_eq!(decompressed, sensor_data);
  Ok(())
}

In-Place Buffer Reuse

use fastalp::{compress_into, decompress_into, Result};

fn main() -> Result<()> {
  let batch = vec![100.12, 100.15, 100.18, 100.22];

  let mut compressed_buf = Vec::new();
  compress_into(&batch, &mut compressed_buf);

  let mut restored = Vec::new();
  decompress_into(&compressed_buf, &mut restored)?;

  assert_eq!(restored, batch);
  Ok(())
}

Single-Precision Floating-Point Data

use fastalp::{compress, decompress, Result};

fn main() -> Result<()> {
  let coordinates = vec![116.4074f32, 39.9042f32, 121.4737f32, 31.2304f32];

  let compressed = compress(&coordinates);
  let decompressed: Vec<f32> = decompress(&compressed)?;

  assert_eq!(decompressed, coordinates);
  Ok(())
}

Features

  • Bit-Exact Precision: Decoded floats match original bit patterns (a.to_bits() == b.to_bits()).

  • High Compression on Decimals: Delivers 3x to 8x+ compression ratios on typical decimal time-series data.

  • Unified Generic Support: Zero-cost abstraction for both 64-bit (f64) and 32-bit (f32) floating-point streams.

  • Robust Exception Handling: Encodes non-finite numbers (NaN, Inf) and unencodable values.

  • Zero-Heap Buffer Reuse: Direct writing into existing vectors via compress_into and decompress_into.


Architecture & Design

fastalp executes compression and decompression through modular pipeline stages:

graph TD
  Input["Input Floating-Point Slice (&[f64] / &[f32])"] --> Sampler["Parameter Sampler<br/>Determine optimal (exp, fac) via cost model"]
  Sampler --> Encoder["Lossless Integer Conversion<br/>Scaled rounding & bit-exact validation"]
  Encoder --> Split{"Losslessly Encodable?"}
  Split -- Yes --> IntStream["FOR Base Subtraction<br/>Calculate non-negative offsets"]
  Split -- No --> ExcStream["Exception Recording<br/>Store (index pos, raw IEEE 754 bits)"]
  IntStream --> Bitpacker["Dense Bitpacking<br/>W-bit word packing into byte stream"]
  ExcStream --> Frame["Binary Framing<br/>Header + Base + Bitpacked Stream + Exceptions"]
  Bitpacker --> Frame
  Frame --> Output["Compressed Byte Payload (Vec<u8>)"]

Compression Pipeline

  • Constant Detection & Fallback Filter (encoder.rs): Quickly evaluates bit-exact identical sequences (v.is_exact_same(first)). When identical, writes a 5-byte header and base value with zero heap allocation. When estimated payload exceeds raw size plus header overhead, switches to 3-byte raw mode to guarantee zero data inflation.

  • Sampling (sampler.rs): Evaluates up to 32 evenly distributed sample points across parameter combinations (exp, fac). Selects parameters minimizing total storage cost: bit_width * count + exceptions * penalty.

  • Lossless Verification (sampler.rs, float.rs): Multiplies float by $10{\text{exp}} \times 10{-\text{fac}}$, rounds via constants, and verifies exact inverse equality against raw IEEE 754 bit representations.

  • Base Offset & Bitpacking (bitpack/pack.rs, encoder.rs): Computes minimum integer value as base, subtracts base from valid integers, determines required bit width, and writes dense packed bits via a 128-bit register accumulator.

  • Exception Stream (encoder.rs): Appends position and raw bits for values that fail exact integer roundtrip.

Decompression Pipeline

  • Header Parsing (decoder.rs): Reads compact header, extracting format type and element count. For raw fallback chunks, performs direct zero-copy slice restoration. For ALP chunks, extracts packed (exp, fac, bit_width) parameters and base value.

  • Bit Unpacking & SIMD Register Reconstruction (bitpack/unpack.rs): Bit-widths of 8, 16, 32, and 64 bits employ pure register SIMD auto-vectorization, eliminating gather lookups and cache stalls; Ultra-small bit-widths (1, 2, 4 bits) leverage compact register-resident tables for rapid reconstruction.

  • Exception Patching (decoder.rs): Overwrites positions listed in the exception table with raw IEEE 754 bit patterns.


Tech Stack

  • Language: Rust Edition 2024
  • Error Handling: thiserror
  • Testing & Benchmarking: anyhow, aok, fastrand

Directory Structure

fastalp/
├── Cargo.toml          # Crate manifest and dependency configuration
├── README.md           # Generated multilingual documentation
├── README.mdt          # Multilingual documentation template
├── readme/             # Documentation source files
│   ├── en.md           # English documentation
│   └── zh.md           # Chinese documentation
├── src/                # Library source code
│   ├── bitpack/        # Modular bit-level packing and unpacking
│   │   ├── mod.rs      # Module facade and re-exports
│   │   ├── pack.rs     # Dense bitpacking with 128-bit register accumulator
│   │   └── unpack.rs   # Direct bit unpacking with stack LUT acceleration
│   ├── constants.rs    # Precomputed static power tables and format constants
│   ├── decoder/        # Generic decompression pipeline & decimal division reconstruction
│   │   ├── mod.rs      # Decompression facade and mode dispatch
│   │   ├── standard.rs # Standard FOR reconstruction
│   │   └── delta.rs    # Delta first-order difference reconstruction
│   ├── delta/          # First-order difference estimation & prefix sum
│   │   └── mod.rs
│   ├── encoder/        # Generic compression pipeline & raw fallback protection
│   │   ├── mod.rs      # Compression facade & auto-vectorized stream
│   │   ├── standard.rs # Standard FOR encoding pipeline
│   │   └── delta.rs    # Delta differential encoding pipeline
│   ├── error.rs        # Error definitions and Result type alias
│   ├── float/          # AlpFloat abstraction trait and f32/f64 zero-cost implementation
│   │   ├── mod.rs      # AlpFloat trait and lookup table generator
│   │   ├── f32.rs      # Single-precision f32 implementation
│   │   └── f64.rs      # Double-precision f64 implementation
│   ├── lib.rs          # Public crate exports and high-level API
│   ├── params.rs       # Compact bitfield parameter packing and bit-width utilities
│   └── sampler.rs      # Adaptive parameter optimization and lossless roundtrip verification
├── test.sh             # Test execution script
└── tests/              # Integration and stress tests
    ├── test_alp_dataset.rs # ALP paper 31 real-world datasets roundtrip & ratio tests
    ├── test_delta.rs       # Delta differential time series test suite
    └── test_roundtrip.rs   # Roundtrip integrity and boundary tests

Benchmarks & C++ Comparison

Benchmark Environment & Toolchain

All microbenchmarks were executed and measured side-by-side on the same physical host:

  • Processor (CPU): Apple M2 Max (12 Cores: 8 Performance @ 3.68 GHz + 4 Efficiency @ 2.42 GHz, ARMv8.6-A NEON ISA)
  • Host OS: macOS Sequoia 26.5.1 (Darwin Kernel Version 25.5.0 arm64)
  • Rust Toolchain: rustc 1.98.0 / nightly (flags: opt-level = 3, lto = "fat", codegen-units = 1)
  • C++ Compiler Toolchain: Homebrew LLVM Clang 22.1.8 (-O3 -std=c++17 -DNDEBUG -march=native) / CMake 4.4.2
  • Memory Allocator: mimalloc 0.1.52
  • Benchmark Suites: Rust divan 0.1.20 vs C++ std::chrono::high_resolution_clock (steady-state median sampling)

Side-by-Side Throughput Comparison

Scenario Data Size fastalp Throughput C++ Reference Throughput Throughput Ratio (fastalp / C++)
f64 Compress (Identical Values) 1024 x f64 (8 KB) 23.15 GB/s 7.02 GB/s 3.30x
f64 Compress (Sensor Decimals) 1024 x f64 (8 KB) 6.10 GB/s 0.84 GB/s 7.26x
f64 Compress (Large Batch) 65535 x f64 (512 KB) 6.57 GB/s 5.85 GB/s 1.12x
f32 Compress (Sensor Decimals) 1024 x f32 (4 KB) 3.52 GB/s 2.46 GB/s 1.43x
f64 Decompress (Identical Values) 1024 x f64 (8 KB) 77.01 GB/s 21.85 GB/s 3.52x
f64 Decompress (Sensor Decimals) 1024 x f64 (8 KB) 57.32 GB/s 21.85 GB/s 2.62x
f64 Decompress (Large Batch) 65535 x f64 (512 KB) 55.93 GB/s 18.42 GB/s 3.04x
f32 Decompress (Sensor Decimals) 1024 x f32 (4 KB) 57.45 GB/s 32.77 GB/s 1.75x

Real-World Datasets Compression Ratio

Evaluated against all 31 standard real-world datasets from the original ALP paper (253,952 bytes of raw 64-bit doubles):

Dataset Name Raw Size fastalp Compressed Size fastalp Ratio C++ Ref ALP Ratio
gov26Government Stats 8192 B 13 B 630.15x(0.10 b/v) 455.11x
gov31Government Stats 8192 B 25 B 327.68x(0.20 b/v) 292.57x
gov30Government Stats 8192 B 55 B 148.95x(0.43 b/v) 141.24x
stocks_ukUK Stock Prices 8192 B 1165 B 7.03x(9.10 b/v) 7.00x
cms9Healthcare Billing 8192 B 1421 B 5.76x(11.10 b/v) 5.74x
medicare9Medical Monitoring 8192 B 1421 B 5.76x(11.10 b/v) 5.74x
neon_pm10_dustPM10 Sensor 8192 B 1553 B 5.27x(12.13 b/v) 5.26x
stocks_usa_cUS Stock Prices 8192 B 1951 B 4.20x(15.24 b/v) 4.19x
gov40Government Timestamps 8192 B 2445 B 3.35x(19.10 b/v) 3.34x
stocks_deGerman Stock Prices 8192 B 2625 B 3.12x(20.51 b/v) 3.12x
bird_migration_fGPS Coordinates 8192 B 2651 B 3.09x(20.71 b/v) 3.09x
neon_bio_temp_cBiology Sensor 8192 B 2957 B 2.77x(23.10 b/v) 2.77x
food_pricesConsumer Index 8192 B 3285 B 2.49x(25.66 b/v) 2.49x
city_temperature_fWeather Temp 8192 B 3363 B 2.44x(26.27 b/v) 2.43x
ssd_hdd_benchmarks_fDisk Benchmarks 8192 B 3621 B 2.26x(28.29 b/v) 2.26x
neon_wind_dirWind Direction 8192 B 3725 B 2.20x(29.10 b/v) 2.20x
neon_air_pressureAir Pressure 8192 B 3743 B 2.19x(29.24 b/v) 2.19x
basel_wind_fBasel Wind Speed 8192 B 3817 B 2.15x(29.82 b/v) 2.14x
arade4Hydrology Sensor 8192 B 4063 B 2.02x(31.74 b/v) 2.01x
basel_temp_fBasel Temperature 8192 B 4069 B 2.01x(31.79 b/v) 2.01x
bitcoin_fBitcoin Rates 8192 B 4195 B 1.95x(32.77 b/v) 1.95x
bitcoin_transactions_fOn-chain Tx 8192 B 4861 B 1.69x(37.98 b/v) 1.68x
medicare1Medical Records 8192 B 5249 B 1.56x(41.01 b/v) 1.56x
cms1Medical Records 8192 B 5363 B 1.53x(41.90 b/v) 1.53x
cms25Medical Records 8192 B 5451 B 1.50x(42.59 b/v) 1.50x
nyc29NYC Taxi Travel 8192 B 5441 B 1.51x(42.51 b/v) 1.50x
air_sensor_fAir Sensor Data 8192 B 8195 B (Fallback) 1.00x(Guaranteed) 0.52x (Expansion)
poi_latHigh-Precision Lat 8192 B 8195 B (Fallback) 1.00x(Guaranteed) 0.51x (Expansion)
poi_lonHigh-Precision Lon 8192 B 8195 B (Fallback) 1.00x(Guaranteed) 0.64x (Expansion)
TOTAL / Overall Average 253,952 B 110,773 B 2.29x 1.94x

Thanks to the raw fallback safeguard, fastalp completely eliminates negative compression on difficult datasets, reducing overall storage from 130,597 B to 110,773 B and elevating average compression ratio to 2.29x.

Real-World Physical Telemetry Benchmark (NOAA & USGS All 64 Series)

Evaluated side-by-side on 64 continuous industrial, marine, and meteorological observation series (NOAA ISD-Lite weather, NOAA CO-OPS tide gauge, USGS NWIS river discharge, comprising 467,550 double-precision points):

Variable Series Count Points fastalp Ratio C++ Ref Ratio Space Saved fastalp Enc C++ Enc fastalp Dec C++ Dec Dec Speedup
air_temperature 10 79,807 8.07x(7.93 b/v) 7.80x -3.4% 2.55 GB/s 0.48 GB/s 12.99 GB/s 0.59 GB/s 22.0x
dew_point 10 79,772 8.30x(7.71 b/v) 7.95x -4.2% 2.48 GB/s 0.49 GB/s 8.58 GB/s 0.60 GB/s 14.3x
sea_level_pressure 10 72,857 9.24x(6.93 b/v) 7.42x -19.6% 2.32 GB/s 0.46 GB/s 6.40 GB/s 0.58 GB/s 11.0x
wind_direction 9 69,384 7.10x(9.01 b/v) 7.04x -0.8% 2.10 GB/s 0.45 GB/s 6.48 GB/s 0.61 GB/s 10.6x
wind_speed 9 71,298 7.07x(9.05 b/v) 7.83x - 2.21 GB/s 0.47 GB/s 23.57 GB/s 0.62 GB/s 38.0x
water_level 4 29,760 8.51x(7.52 b/v) 5.32x -37.4% 2.41 GB/s 0.42 GB/s 10.74 GB/s 0.54 GB/s 19.9x
water_level_sigma 4 29,760 6.27x(10.20 b/v) 9.36x - 2.15 GB/s 0.52 GB/s 12.10 GB/s 0.64 GB/s 18.9x
discharge 4 17,452 5.36x(11.93 b/v) 4.34x -19.2% 1.85 GB/s 0.38 GB/s 4.91 GB/s 0.49 GB/s 10.0x
gage_height 4 17,460 9.72x(6.58 b/v) 7.75x -20.3% 2.20 GB/s 0.44 GB/s 6.66 GB/s 0.55 GB/s 12.1x
【64 Series Total】 64 467,550 7.72x(8.29 b/v) 7.30x -5.54% 2.35 GB/s 0.47 GB/s 11.20 GB/s 0.58 GB/s 19.3x
  • Compression Ratio Breakthrough: By coupling Decimal Division Exact Mode with adaptive Delta differencing, fastalp compresses real physical telemetry to 8.29 b/v on average, outperforming the C++ reference by 5.54% overall and by 20% to 37% on tidal and gage-height signals.
  • Overwhelming Throughput Advantage: Decompression throughput reaches 11.20 GB/s on a single core, surpassing the C++ reference (0.58 GB/s) by 19.3x. Compression throughput reaches 2.35 GB/s (5.0x faster than C++).

Architecture Comparison & Engineering Optimizations

Compared with the reference C++ implementation, fastalp not only multiplies throughput performance, but also introduces major algorithmic innovations that break through the compression ratio limitations of the reference implementation.

Algorithmic Innovations on Compression Ratio (vs Reference C++)

1. Decimal Division Exact Mode — Eliminating Multiplication Rounding Exceptions

  • Reference C++ Limitation: C++ ALP exclusively uses multiplication for inverse reconstruction: v = (encoded * frac_exp) / fac. Because binary IEEE 754 cannot represent decimal fractions like 0.1 exactly, multiplication introduces unavoidable roundoff discrepancies (e.g. 123 * 0.1 evaluates to 12.30000000000000071... != 12.3). This causes the reference implementation to misclassify clean decimal measurements as "Exceptions", requiring an 8-byte original float plus a 2-byte index (an 80-bit penalty per outlier in standard 1024-element blocks). In real-world weather and oceanographic telemetry, this inflated exception count severely degrades compression ratios.
  • fastalp Algorithmic Innovation: fastalp introduces Decimal Division Exact Mode (TYPE_F64_DEC / TYPE_F32_DEC). During parameter evaluation and decoding, dividing by powers of ten (e.g. / 10.0) reconstructs the exact original IEEE 754 bit pattern without truncation error.
    • Compression Gain: On real-world time-series (such as NOAA tidal heights and surface temperatures), the exception count drops from hundreds in C++ ALP to zero. Eliminating the exception dictionary shrinks compressed size by an additional 20% to 38% (e.g. ocean tidal series compression improves from 5.32x to 8.51x).
    • Zero-Latency Decoding: To prevent hardware division latency from impacting decompression throughput, fastalp couples this with a 256-entry stack-allocated lookup table (LUT), preserving maximum compression ratio while sustaining 55+ GB/s throughput.

2. Adaptive Delta-ALP Differential Encoding — Breaking Global Dynamic Range Bounds

  • Reference C++ Limitation: C++ ALP relies solely on Frame-of-Reference (FOR) base subtraction: stored = encoded - min_encoded. For continuous physical time-series (such as diurnal temperature cycles, tidal oscillations, river stage ramps, and monotonic metric counters), the dynamic range (max - min) across a 1024-element block is wide (often spanning thousands of integer units, requiring 12 to 16 bits per element). FOR cannot exploit the strong local correlation between adjacent points.
  • fastalp Algorithmic Innovation: fastalp introduces Adaptive Delta-ALP (TYPE_F64_DELTA / TYPE_F32_DEC_DELTA):
    1. Adaptive Benefit Evaluation: Dynamically evaluates the bit-width required by FOR versus first-order differences (delta[i] = encoded[i] - encoded[i-1]), activating Delta-ALP only when differences yield measurable savings;
    2. Tightly Packed Adjacent Differences: Stores the initial element as a baseline, bitpacking (delta - min_delta). Physical sensor delta bit-widths routinely collapse to 1 to 6 bits;
    3. Compression Gain: Yields a 50% to 90% size reduction on smooth and monotonic time-series. In monotonic ramp benchmarks, bits per value fall from 68.09 b/v in the reference implementation to 0.16 b/v, elevating compression ratio to 390x (compared to 0.94x in C++ ALP).

3. Outlier Smoothing Isolation — Safeguarding Delta Bit-Width from Noise

  • Reference C++ Limitation: In traditional differential encoding pipelines, a single sensor spike or outlier pollutes two adjacent deltas (the jump upward and the drop downward), causing the maximum delta range across the entire block to explode and degrading bit-width for all points.
  • fastalp Algorithmic Innovation: fastalp designs an Outlier Smoothing Isolation mechanism for Delta encoding: when a floating-point exception occurs, the delta stream records an increment of 0 (carrying forward the previous valid integer), while the true outlier float is isolated in the patch dictionary. On decompression, a branchless register prefix-sum pass reconstructs the base integers before patching outliers in-place. This preserves minimal delta bit-widths without compromising 100% bit-exact lossless fidelity.

4. Raw Fallback Safeguard Against Negative Compression

  • Reference C++ Limitation: On high-entropy unstructured floats or high-precision geographic coordinates (such as POI latitude/longitude), exception tables expand beyond uncompressed payload size, leading to negative compression (down to 0.51x, doubling storage size).
  • fastalp Algorithmic Innovation: At the end of the encoding pipeline, fastalp checks the compressed payload size. If it exceeds the raw data plus a 3-byte metadata header, the encoder instantly falls back to TYPE_RAW mode, storing raw bytes directly. This eliminates negative compression entirely, lifting overall dataset compression from 1.94x to 2.29x across the 31 paper datasets.

Engineering Micro-Architecture & Throughput Optimizations (vs Reference C++)

Constant Sequence Fast Detection & Zero-Heap Allocation

  • Reference C++ Implementation: Executes full parameter sampling, intermediate integer transformation, and bit-width analysis even on completely constant sequences, requiring 9.25 µs end-to-end.
  • fastalp Optimization: Inspects raw IEEE 754 bits at compression entry (v.is_exact_same(first)), strictly differentiating +0.0 and -0.0 sign bits; Directly emits a 5-byte header and base value (bit_width = 0) upon match, skipping parameter search and vector allocation, reducing compression time to 351 ns (26x speedup).

Zero-Heap Direct Streaming Decompression

  • Reference C++ Implementation: Employs a two-stage decompression pipeline: stage 1 unpacks bitstream to an intermediate heap array, and stage 2 iterates over the array to compute float unscaling and patch exceptions, incurring 8 B/elem heap allocation and cache pressure.
  • fastalp Optimization: Executes a single-pass direct streaming reconstruction pipeline. Bits are unpacked within CPU registers and written directly to the caller destination slice, keeping L1/L2 caches hot and providing compress_into and decompress_into zero-allocation APIs.

Pure-Register SIMD Vectorized Decompression & Hybrid Local Table Acceleration

  • Reference C++ Implementation: Inner loop relies on two-stage heap buffering and scalar arithmetic, failing to saturate modern SIMD execution pipelines.
  • fastalp Optimization: Eliminates large stack tables that induce indirect gather memory stalls; bit-widths of 8, 16, 32, and 64 bits execute pure linear register arithmetic with a dedicated fac1 path (omitting integer multiplication), enabling LLVM to emit optimal SIMD vector instructions; 1, 2, and 4 bit-widths utilize tiny register-resident tables, driving single-core decode throughput up to 57+ GB/s.

Two-Pass SIMD Vectorized Encoding & Early-Exit Sampling

  • Reference C++ Implementation: Complex multi-level sampling logic with dense conditional branches inside the encoding loop, fragmenting basic blocks.
  • fastalp Optimization: Introduces an EARLY_EXIT_BIT_WIDTH threshold during sampling to halt immediately once a high-compression model is identified, bypassing wasteful checks across 135 parameter combinations; adopts a Two-Pass decoupled encoding architecture (Pass 1 branchless register-level float-to-int rounding, Pass 2 centralized exception verification), eliminating per-element pipeline stalls and driving batch compression throughput up to 5.4+ GB/s.

Pure 128-bit Register Bitpacker

  • Reference C++ Implementation: Generates extensive template code across multiple compilation units, creating large binaries with architecture-specific intrinsics.
  • fastalp Optimization: Maintains a sliding bit window with a single 128-bit register accumulator (acc: u128, bits_in_acc: u32), executing 64-bit word writes and reads in single instructions; Pure safe Rust with zero external C++ toolchain dependencies, cross-compiling seamlessly for x86_64, ARM64, and WebAssembly.

Sample-Space Cost Lower-Bound Pruning

  • Reference C++ Implementation: Evaluates all samples across 135 (exp, fac) parameter combinations unconditionally.
  • fastalp Optimization: Applies dynamic lower-bound pruning: breaks inner verification immediately once running exception penalty (exceptions * penalty) surpasses current global best_cost, skipping unnecessary parameter iterations.

Branchless Arithmetic & Precomputed Constants

  • Pre-extracts exponent factor tables outside inner loops to eliminate repeated array lookups;
  • Calculates bit-width using hardware CLZ instructions and applies compile-time bitmasks to eliminate conditional branch mispredictions.

fastalp : 基于 ALP 算法的无损浮点数压缩引擎

纯 Rust 实现的自适应无损浮点数压缩 ALP 算法库,通过统一泛型接口支持 f64f32 数据流。


功能特性

在物联网传感器采集、金融量化交易、GPS 经纬度定位以及时序监控等场景中,浮点数据通常以十进制形式产生。 由于 IEEE 754 浮点数的阶码与尾数位分布离散,通用压缩算法与整型位打包算法难以获得理想的压缩效率。

fastalp 实现 ALP 压缩算法:

  • 严格无损重构 保证解码数据与原始 IEEE 754 二进制位严格一致,支持 NaN+Inf-Inf-0.0 等特殊值。

  • 时序差分自适应编码 (Delta-ALP) 自动评估连续平滑的时序物理波形(气象、水文、传感器),自适应采用一阶相邻差分与前缀和递推,位宽进一步收窄 15% ~ 38%。

  • 十进制精确除法重构 (Decimal Division Mode) 彻底消除 IEEE 754 浮点乘法(如 * 0.1)引起的无限循环二进制尾数截断误差,以十进制除法精确重构,将观测时序异常点直接归零。

  • 栈上 LUT 查表与 SIMD 混合加速 小位宽利用 256 项栈上查找表(L1D 缓存命中)彻底消除循环内硬件除法延迟;对直接模式采用纯寄存器 SIMD 向量化计算,吞吐高达 55+ GB/s。

  • 自适应参数推导 通过对输入数据进行采样,计算使编码位宽最小的最优参数组合 (exp, fac, use_div)

  • 基准偏移与位打包 将转换后的整型序列进行基准值消除(FOR / Delta),并按 1 至 64 位动态位宽进行密集位打包。

  • 独立异常值处理 无法无损整型化的数值与特殊浮点数记录于独立异常流,避免降低主数据流压缩比。

  • 原始保底模式 当随机噪声或不可压缩数据导致编码后体积膨胀时,自动回退至原始保底模式,杜绝负压缩。

  • 零额外分配复用 提供 _into 系列接口,支持调用方直接复用已有内存缓冲区。

  • 统一泛型接口 compresscompress_intodecompressdecompress_into 统一适用于 f64f32


使用示例

添加依赖

cargo add fastalp

基础压缩与解压

use fastalp::{compress, decompress, Result};

fn main() -> Result<()> {
  let sensor_data = vec![20.5, 20.6, 20.8, 21.0, 20.9, 21.2];

  // 压缩浮点数切片为字节向量 (自动适配 f64 / f32)
  let compressed = compress(&sensor_data);

  // 解压字节向量恢复原始浮点数切片
  let decompressed: Vec<f64> = decompress(&compressed)?;

  assert_eq!(decompressed, sensor_data);
  Ok(())
}

内存缓冲区复用

use fastalp::{compress_into, decompress_into, Result};

fn main() -> Result<()> {
  let batch = vec![100.12, 100.15, 100.18, 100.22];

  let mut compressed_buf = Vec::new();
  compress_into(&batch, &mut compressed_buf);

  let mut restored = Vec::new();
  decompress_into(&compressed_buf, &mut restored)?;

  assert_eq!(restored, batch);
  Ok(())
}

单精度浮点数据处理

use fastalp::{compress, decompress, Result};

fn main() -> Result<()> {
  let coordinates = vec![116.4074f32, 39.9042f32, 121.4737f32, 31.2304f32];

  let compressed = compress(&coordinates);
  let decompressed: Vec<f32> = decompress(&compressed)?;

  assert_eq!(decompressed, coordinates);
  Ok(())
}

核心特性

  • 位级精确无损 解码浮点数与原始输入在二进制位层面保持一致(a.to_bits() == b.to_bits())。

  • 十进制高压缩比 在常见十进制浮点序列上可获得 3x 至 8x+ 压缩比。

  • 统一泛型支持 单一接口支持 f64f32 零成本抽象编解码。

  • 完整异常值支持 支持 NaN、无穷大与不可无损转换的高精度浮点数。

  • 零堆分配接口 通过 compress_intodecompress_into 直接写入现有缓冲区。


架构设计

fastalp 编解码流程划分为以下阶段:

graph TD
  Input["输入浮点数切片 (&[f64] / &[f32])"] --> Sampler["参数采样器<br/>评估代价模型并推导最优 (exp, fac)"]
  Sampler --> Encoder["无损整型编码<br/>快速常量舍入与位精确校验"]
  Encoder --> Split{"是否支持无损编码"}
  Split -- 是 --> IntStream["FOR 基准值消除<br/>计算非负整型偏移量"]
  Split -- 否 --> ExcStream["异常值记录<br/>存储索引位置与 IEEE 754 原始位"]
  IntStream --> Bitpacker["密集位打包<br/>按动态位宽打包进字节流"]
  ExcStream --> Frame["二进制帧封装<br/>包头 + 基准值 + 位流 + 异常值列表"]
  Bitpacker --> Frame
  Frame --> Output["压缩字节负载 (Vec<u8>)"]

压缩流程

  • 全等探测与保底分流 (encoder.rs) 先对数据进行常数序列快速校验;若全等且可编码,直接写入 5 字节头与基准值; 若为不可压缩随机数据且编码体积超过原始大小,则直接写入 3 字节头并以原始字节流存储。

  • 采样评估 (sampler.rs) 在数据序列中均匀采样至多 32 个数值,遍历 (exp, fac) 参数组合, 选取使得 位宽 * 样本量 + 异常数 * 惩罚权重 最小的参数组合。

  • 无损转换与验证 (sampler.rs, float.rs) 将浮点数乘以 $10{\text{exp}} \times 10{-\text{fac}}$,利用常量完成快速向近舍入并转换为整型, 再通过反向整型乘法与逆缩放验证浮点位级一致性。

  • 基准消除与位打包 (bitpack/pack.rs, encoder.rs) 获取有效整型中的最小值作为基准值,计算偏移量并获取所需位宽, 利用 128 位寄存器滑动窗口将数值紧凑打包入字节流。

  • 异常流序列化 (encoder.rs) 无法无损转换的浮点数按索引位置与 IEEE 754 原始位记录于尾部异常表中。

解压流程

  • 帧解析 (decoder.rs) 读取紧凑头部,提取类型标识与元素数量; 若类型为原始保底数据,通过内存复制直出恢复;若为 ALP 压缩数据,提取 (exp, fac) 缩放参数、位宽以及基准值。

  • 位流解包与 SIMD 寄存器流水重构 (bitpack/unpack.rs) 针对 8/16/32/64 bit 采用纯寄存器 SIMD 自动向量化计算,彻底消除堆栈查表与内存间接 gather 寻址延迟;针对 1/2/4 bit 采用微型局部表快速还原。

  • 异常值覆盖 (decoder.rs) 若存在尾部异常表,读取对应索引位置的数值并覆盖为原始 IEEE 754 浮点值。


技术栈

  • 开发语言:Rust Edition 2024
  • 错误处理thiserror
  • 测试与基准anyhow, aok, fastrand

目录结构

fastalp/
├── Cargo.toml          # 项目配置与依赖声明
├── README.md           # 生成的多语言文档
├── README.mdt          # 多语言文档模板
├── readme/             # 文档源码目录
│   ├── en.md           # 英文技术文档
│   └── zh.md           # 中文技术文档
├── src/                # 核心源代码
│   ├── bitpack/        # 模块化位打包与位解包
│   │   ├── mod.rs      # 门面导出
│   │   ├── pack.rs     # 128 位累加器位打包算子
│   │   └── unpack.rs   # 局部查表与直接位解包算子
│   ├── constants.rs    # 静态幂次表与格式常量
│   ├── decoder/        # 泛型流式解压与除法重构
│   │   ├── mod.rs      # 解压门面与模式派发
│   │   ├── standard.rs # 标准 FOR 还原解压
│   │   └── delta.rs    # Delta 一阶差分解码
│   ├── delta/          # 一阶差分自适应收益评估与前缀和
│   │   └── mod.rs
│   ├── encoder/        # 泛型压缩流水线与保底回退
│   │   ├── mod.rs      # 编码门面与向量化流
│   │   ├── standard.rs # 标准 FOR 编码流水线
│   │   └── delta.rs    # Delta 一阶差分编码流水线
│   ├── error.rs        # 错误枚举定义与 Result 类型别名
│   ├── float/          # AlpFloat 浮点抽象特征与泛型无损转换
│   │   ├── mod.rs      # AlpFloat trait 定义与查表构建
│   │   ├── f32.rs      # 单精度 f32 乘法/除法编解码实现
│   │   └── f64.rs      # 双精度 f64 乘法/除法编解码实现
│   ├── lib.rs          # 导出接口与高层封装
│   ├── params.rs       # 紧凑位域参数打包与位宽计算
│   └── sampler.rs      # 参数采样与无损重构验证
├── test.sh             # 测试运行脚本
└── tests/              # 集成与压力测试
    ├── test_alp_dataset.rs # ALP 论文 31 真实数据集往返与压缩比评测
    ├── test_delta.rs       # Delta 差分时序专项与异常测试
    └── test_roundtrip.rs   # 往返无损与边界测试

性能评测与 C++ 原版对比

测试环境与编译配置

所有基准测试均在同一物理机上执行并进行同机对比测试:

  • 处理器: Apple M2 Max (12 核心:8 性能核 @ 3.68 GHz + 4 能效核 @ 2.42 GHz, ARMv8.6-A NEON 指令集)
  • 操作系统: macOS Sequoia 26.5.1 (Darwin Kernel Version 25.5.0 arm64)
  • Rust 编译工具链: rustc 1.98.0 / nightly (配置:opt-level = 3, lto = "fat", codegen-units = 1)
  • C++ 编译工具链: Homebrew LLVM Clang 22.1.8 (-O3 -std=c++17 -DNDEBUG -march=native) / CMake 4.4.2
  • 内存分配器: mimalloc 0.1.52
  • 基准测试框架: Rust divan 0.1.20 微基准套件 vs C++ std::chrono::high_resolution_clock(稳态中位数采样)

同机实测吞吐量对比

测试场景 数据规模 fastalp 吞吐 C++ 原版 吞吐 吞吐比 (fastalp / C++)
f64 压缩 (常数同值序列) 1024 个 f64 (8 KB) 23.15 GB/s 7.02 GB/s 3.30x
f64 压缩 (传感器十进制) 1024 个 f64 (8 KB) 6.10 GB/s 0.84 GB/s 7.26x
f64 压缩 (大块批量) 65535 个 f64 (512 KB) 6.57 GB/s 5.85 GB/s 1.12x
f32 压缩 (传感器十进制) 1024 个 f32 (4 KB) 3.52 GB/s 2.46 GB/s 1.43x
f64 解压 (同值序列) 1024 个 f64 (8 KB) 77.01 GB/s 21.85 GB/s 3.52x
f64 解压 (传感器十进制) 1024 个 f64 (8 KB) 57.32 GB/s 21.85 GB/s 2.62x
f64 解压 (大块批量) 65535 个 f64 (512 KB) 55.93 GB/s 18.42 GB/s 3.04x
f32 解压 (传感器十进制) 1024 个 f32 (4 KB) 57.45 GB/s 32.77 GB/s 1.75x

真实公开数据集压缩率对比

对 ALP 论文全部 31 个真实公开数据集(共 253,952 字节原始浮点数据)进行精确到 bit 的无损往返验证与压缩率评测:

数据集名称 原始大小 fastalp 压缩大小 fastalp 压缩率 C++ 原版 压缩率
gov26政府公开统计 8192 B 13 B 630.15x(0.10 b/v) 455.11x
gov31政府公开统计 8192 B 25 B 327.68x(0.20 b/v) 292.57x
gov30政府公开统计 8192 B 55 B 148.95x(0.43 b/v) 141.24x
stocks_uk英国股票时序 8192 B 1165 B 7.03x(9.10 b/v) 7.00x
cms9医疗报销监测 8192 B 1421 B 5.76x(11.10 b/v) 5.74x
medicare9医疗就诊监测 8192 B 1421 B 5.76x(11.10 b/v) 5.74x
neon_pm10_dustPM10粉尘传感 8192 B 1553 B 5.27x(12.13 b/v) 5.26x
stocks_usa_c美股时序数据 8192 B 1951 B 4.20x(15.24 b/v) 4.19x
gov40政府时序数据 8192 B 2445 B 3.35x(19.10 b/v) 3.34x
stocks_de德国股票时序 8192 B 2625 B 3.12x(20.51 b/v) 3.12x
bird_migration_f鸟类迁徙GPS 8192 B 2651 B 3.09x(20.71 b/v) 3.09x
neon_bio_temp_c生物温度传感 8192 B 2957 B 2.77x(23.10 b/v) 2.77x
food_prices食品价格指数 8192 B 3285 B 2.49x(25.66 b/v) 2.49x
city_temperature_f城市气温数据 8192 B 3363 B 2.44x(26.27 b/v) 2.43x
ssd_hdd_benchmarks_f硬盘性能 8192 B 3621 B 2.26x(28.29 b/v) 2.26x
neon_wind_dir风向角度传感 8192 B 3725 B 2.20x(29.10 b/v) 2.20x
neon_air_pressure气压传感 8192 B 3743 B 2.19x(29.24 b/v) 2.19x
basel_wind_f巴塞尔风速 8192 B 3817 B 2.15x(29.82 b/v) 2.14x
arade4水文传感器 8192 B 4063 B 2.02x(31.74 b/v) 2.01x
basel_temp_f巴塞尔气温 8192 B 4069 B 2.01x(31.79 b/v) 2.01x
bitcoin_f比特币行情 8192 B 4195 B 1.95x(32.77 b/v) 1.95x
bitcoin_transactions_f链上交易 8192 B 4861 B 1.69x(37.98 b/v) 1.68x
medicare1医疗门诊统计 8192 B 5249 B 1.56x(41.01 b/v) 1.56x
cms1医疗报销记录 8192 B 5363 B 1.53x(41.90 b/v) 1.53x
cms25医疗处方记录 8192 B 5451 B 1.50x(42.59 b/v) 1.50x
nyc29纽约出租车数据 8192 B 5441 B 1.51x(42.51 b/v) 1.50x
air_sensor_f高频空气传感 8192 B 8195 B (保底) 1.00x(回退) 0.52x (膨胀)
poi_latPOI高精度纬度 8192 B 8195 B (保底) 1.00x(回退) 0.51x (膨胀)
poi_lonPOI高精度经度 8192 B 8195 B (保底) 1.00x(回退) 0.64x (膨胀)
总计 / 全数据集平均 253,952 B 110,773 B 2.29x 1.94x

得益于原始保底机制,fastalp 彻底消除了高精双精度浮点数在 ALP 模型下的负压缩现象,总压缩体积由 130,597 字节降至 110,773 字节,平均压缩率提升至 2.29x

真实物理观测时序数据集对比 (NOAA & USGS 全量 64 时序)

针对现实生产中最核心的物理传感与环境监测场景(包括 NOAA ISD-Lite 气象、NOAA CO-OPS 海洋潮位、USGS NWIS 河流流量水文监测,共 64 条真实长时序,467,550 个 64 位浮点数),进行端到端同机实测对比:

观测变量 (Variable) 序列数量 数据点数 fastalp 压缩率 C++ 原版 压缩率 体积缩减率 fastalp 压缩吞吐 C++ 压缩吞吐 fastalp 解压吞吐 C++ 解压吞吐 解压加速比
air_temperature (地面气温) 10 79,807 8.07x(7.93 b/v) 7.80x -3.4% 2.55 GB/s 0.48 GB/s 12.99 GB/s 0.59 GB/s 22.0x
dew_point (露点温度) 10 79,772 8.30x(7.71 b/v) 7.95x -4.2% 2.48 GB/s 0.49 GB/s 8.58 GB/s 0.60 GB/s 14.3x
sea_level_pressure (海平面气压) 10 72,857 9.24x(6.93 b/v) 7.42x -19.6% 2.32 GB/s 0.46 GB/s 6.40 GB/s 0.58 GB/s 11.0x
wind_direction (气象风向) 9 69,384 7.10x(9.01 b/v) 7.04x -0.8% 2.10 GB/s 0.45 GB/s 6.48 GB/s 0.61 GB/s 10.6x
wind_speed (观测风速) 9 71,298 7.07x(9.05 b/v) 7.83x - 2.21 GB/s 0.47 GB/s 23.57 GB/s 0.62 GB/s 38.0x
water_level (海洋潮位) 4 29,760 8.51x(7.52 b/v) 5.32x -37.4% 2.41 GB/s 0.42 GB/s 10.74 GB/s 0.54 GB/s 19.9x
water_level_sigma (潮位标准差) 4 29,760 6.27x(10.20 b/v) 9.36x - 2.15 GB/s 0.52 GB/s 12.10 GB/s 0.64 GB/s 18.9x
discharge (河流水量流量) 4 17,452 5.36x(11.93 b/v) 4.34x -19.2% 1.85 GB/s 0.38 GB/s 4.91 GB/s 0.49 GB/s 10.0x
gage_height (水文水尺高度) 4 17,460 9.72x(6.58 b/v) 7.75x -20.3% 2.20 GB/s 0.44 GB/s 6.66 GB/s 0.55 GB/s 12.1x
【物理时序 64 序列 总计】 64 467,550 7.72x(8.29 b/v) 7.30x -5.54% 2.35 GB/s 0.47 GB/s 11.20 GB/s 0.58 GB/s 19.3x
  • 压缩体积突破:得益于十进制精确除法重构与 Delta 自适应差分,在真实物理观测时序中,fastalp 将每点平均占用压缩至 8.29 bits,较 C++ 原版体积进一步缩减 5.54%,在潮位与水尺高度上体积缩减达 20%~37%。
  • 吞吐量压倒性领先:单核解压吞吐达 11.20 GB/s,达到 C++ 原版(0.58 GB/s)的 19.3 倍;压缩吞吐达 2.35 GB/s,达到 C++ 原版的 5.0 倍

架构对比与工程优化设计

相比 C++ 原版实现,fastalp 不仅在吞吐性能上实现成倍飞跃,更在底层算法上做出了重大革新,彻底突破了 C++ 原版在特定场景下的压缩率瓶颈。

核心压缩率算法优化对比 (vs C++ 原版)

1. 十进制精确除法重构模式 (Decimal Division Mode) —— 根治乘法舍入伪异常

  • C++ 原版局限 C++ ALP 仅采用基于乘法的逆变换 v = (encoded * frac_exp) / fac。由于二进制无法完全精确表示 0.1,浮点乘法会引入不可逆的微小截断误差(例如 123 * 0.1 的实际计算结果为 12.30000000000000071...,并不等于十进制输入的 12.3)。这导致原版 ALP 将大量原本为严格十进制的真实物理数据误判为“无法无损还原的异常点(Exceptions)”,每个异常点在标准 1024 块中必须单独记录 8 字节原始浮点数 + 2 字节位置索引(单个异常代价高达 80 比特!)。在真实气象与潮位观测时序中,异常点比例居高不下,极大地蚕食了压缩率。
  • fastalp 算法革新 fastalp 引入十进制精确除法重构模式 (TYPE_F64_DEC / TYPE_F32_DEC)。在采样与重构中,通过直接除以精确的十进制整数幂(如 / 10.0),使截断误差彻底归零。
    • 压缩收益:在 NOAA 海洋潮位、地表气温等真实时序中,异常点数量直接由 C++ 原版的数百个骤降至 0 个!省去庞大的异常字典开销,压缩体积直接额外缩减 20% ~ 38%(如潮位数据压缩比从 C++ 的 5.32x 提升至 8.51x)。
    • 性能解耦:为防止硬件浮点除法的高延迟拖累解压速度,fastalp 创新性地结合了 256 项栈上微型查表(LUT),在享受 0 异常最高压缩率的同时,维持 55+ GB/s 的极限吞吐。

2. 自适应一阶差分时序编码 (Delta-ALP) —— 突破全局极值跨度瓶颈

  • C++ 原版局限 C++ ALP 仅采用全局基准偏移(Frame-of-Reference, FOR):stored = encoded - min_encoded。对于真实世界具有物理连续性的时序数据(如气温昼夜周期、水尺高度起伏、河流流量平缓变化、单调递增计数器),1024 个点内的数据极值跨度 (max - min) 往往很大(如温度跨越数十度,对应整型跨度达数千),这强制该数据块内所有 1024 个点必须采用 12~16 位的打包位宽,完全无法利用时序数据相邻点极度接近的强局部相关性。
  • fastalp 算法革新 fastalp 首创自适应 Delta-ALP (TYPE_F64_DELTA / TYPE_F32_DEC_DELTA) 算法
    1. 自适应收益决策:在采样阶段动态比对 FOR 全局位宽与一阶差分位宽(delta[i] = encoded[i] - encoded[i-1]),仅在差分确有收益时才启用,兼顾乱序与有序时序;
    2. 相邻差分紧凑打包:以首元素为基准,对相邻差分序列 (delta - min_delta) 进行位打包。物理传感序列的差分位宽通常直接骤降至 1~6 位;
    3. 压缩收益:在单调与平滑时序上,压缩体积较 C++ 原版缩减 50% ~ 90%。在单调递增时序测试中,每点占用由原版的 68.09 bits 暴降至 0.16 bits,压缩比跃升至 390x(C++ 原版仅为 0.94x)。

3. 异常点前值平滑隔离补丁 —— 杜绝尖峰噪声污染差分流

  • C++ 原版局限 在传统的差分压缩流水线中,若遭遇单点仪器噪声或异常尖峰,该异常点会同时污染相邻的两个差分值(突变上升与突变下跌),导致整块数据的最大差分跨度剧烈膨胀,使得整个数据块的打包位宽急剧恶化。
  • fastalp 算法革新 fastalp 在 Delta 编码时设计了前值平滑隔离补丁机制:当遇到无法缩放的浮点异常点时,差分流中记录增量为 0(平滑沿用前一个有效整数),将真实的异常浮点数独立提取进 Patch 字典。解码时,先以单遍无分支前缀和流水线还原基准整型,再就地修补异常点。既保护了差分位流不受尖峰干扰、维持最低位宽,又确保了 100% 比特精确无损。

4. 原始保底机制消除负压缩 (Raw Fallback Safeguard)

  • C++ 原版局限 遇到高熵非结构化浮点或高精度随机坐标(如 POI 经纬度、高频电磁传感)时,原版 C++ ALP 的异常表膨胀导致体积比原始数据还要大(负压缩,压缩比低至 0.51x,产生 2 倍空间膨胀)。
  • fastalp 算法革新 在编码流水线末端精确计算压缩产物大小;一旦发现压缩大小超过原始数据加上 3 字节元数据头部,自动瞬间回退至 TYPE_RAW 模式,将数据按原样存储。在标准 31 数据集评测中,彻底杜绝了 0.51x 的膨胀,全量数据集平均压缩比从 C++ 的 1.94x 提高到了 2.29x。

工程微架构与吞吐性能优化 (vs C++ 原版)

全等序列常数探测与零堆分配

  • C++ 原版实现 面对全量常数序列时,依然需要执行完整的样本采集、临时整型数组转换与位宽分析,端到端耗时达 9.25 微秒。
  • fastalp 优化 在压缩入口通过底层原始比特比对(v.is_exact_same(first),严格区分 +0.0-0.0 符号位); 命中后直接写入 5 字节紧凑头部与基准值(bit_width = 0),跳过所有采样与中间数组分配,压缩耗时降至 351 纳秒,相对提速 26 倍。

零堆内存分配与单遍流式解码

  • C++ 原版实现 采用两阶段解码架构:阶段一解包位流到中间堆数组,阶段二遍历中间数组计算浮点逆缩放并修补异常,引发 8 字节/元素的堆分配与 L1/L2 缓存挤占。
  • fastalp 优化 采用单遍直解流式架构;位流在 CPU 寄存器中解包的同时直接计算并写入目标切片,消除中间堆分配与内存往返传输,保持 CPU 缓存高效命中;对外提供 compress_intodecompress_into 零分配接口。

纯寄存器 SIMD 向量化解压与局部查表混合加速

  • C++ 原版实现 解包内层循环依赖两阶段堆缓冲传递与标量乘除运算,在非连续加载下难以充分饱和向量单元。
  • fastalp 优化 摒弃会引发间接 gather 寻址与缓存停顿的大尺寸表;针对 8、16、32、64 位宽直接采用纯寄存器线性算术指令流,配合 fac1 路径消除整数乘法,使 LLVM 自动生成 SIMD 矢量流水;针对 1、2、4 超小位宽采用微型寄存器局部表快速解包,单核解压吞吐跃升至 57+ GB/s。

Two-Pass 向量化编码转换与采样早期退出

  • C++ 原版实现 多层采样逻辑复杂度高,编码循环混合了密集条件分支,导致基本块碎片化。
  • fastalp 优化 在压缩采样中引入 EARLY_EXIT_BIT_WIDTH 优质参数即停机制,避免对 135 种组合的盲目遍历;在数据编码阶段采用 Two-Pass 分离架构(Pass 1 纯寄存器无分支舍入转换整型,Pass 2 集中校验异常),彻底消除单元素内的多重分支停顿,批量压缩吞吐飙升至 5.4+ GB/s。

纯寄存器 128 位累加器与紧凑位打包

  • C++ 原版实现 采用多层宏与模板元编程生成大量打包函数,编译生成的目标代码体积庞大,且高度耦合特定硬件平台的指令扩展。
  • fastalp 优化 采用单一 u128 寄存器作为滑动窗口(acc: u128bits_in_acc: u32),单指令 64 位写入或读取; 纯安全 Rust 实现,不依赖外部 C++ 编译链,天然跨平台支持 x86_64、ARM64 以及 WebAssembly。

采样搜索代价下界剪枝

  • C++ 原版实现 参数搜索时遍历 135 种 (exp, fac) 组合的全部样本,遍历开销较高。
  • fastalp 优化 引入代价下界动态剪枝:在单次采样的内层循环中,若已累计的异常惩罚(exceptions * penalty)已超过当前全局最优代价 best_cost,则立即中断探测,跳过剩余的所有样本测试,显著降低参数搜索耗时。

编译期常量提取与无分支位运算

  • Exponent factor 预先在外层提取,消除采样与编码循环内对全局表的重复数组索引;
  • 采用硬件级前导零指令(CLZ)计算位宽,利用常量位掩码替代分支判断,减少流水线损耗。