Module fastlanes 

FastLanes-inspired FOR + bit-packing codec for integer columns.

Frame-of-Reference (FOR): subtract the minimum value from all values, reducing them to small unsigned residuals. Then bit-pack the residuals using the minimum number of bits.

The bit-packing loop is written as simple scalar operations on contiguous arrays, which LLVM auto-vectorizes to AVX2/AVX-512/NEON/WASM-SIMD without explicit intrinsics. This is the FastLanes insight: structured scalar code that the compiler vectorizes, portable across all targets.

Wire format:

[4 bytes] total value count (LE u32)
[2 bytes] block count (LE u16)
For each block:
  [2 bytes] values in this block (LE u16, max 1024)
  [1 byte]  bit width (0-64)
  [8 bytes] min value / reference (LE i64)
  [N bytes] bit-packed residuals

Block size: 1024 values. Last block may be smaller.

Structs

BlockIterator

Iterator that decodes one 1024-row block at a time, tracking byte offsets internally. Avoids re-scanning headers for sequential access.

Functions

bit_width_for_range

Compute the minimum number of bits needed to represent the range of values.

block_byte_offsets

Compute byte offsets for each block in an encoded stream.

block_count

Number of blocks in an encoded FastLanes stream.

decode

Decode FOR + bit-packed bytes back to i64 values.

decode_block_range

Decode a range of blocks [start_block..end_block) from encoded data.

decode_single_block

Decode a single block by index without decoding the entire stream.

encode

Encode a slice of i64 values using FOR + bit-packing.