Struct OutOfLineBitpacking 

pub struct OutOfLineBitpacking { /* private fields */ }

A transparent compressor that bit packs data

In order for the encoding to be transparent we must have a fixed bit width across the entire array. Chunking within the buffer is not supported. This means that we will be slightly less efficient than something like the mini-block approach.

This was an interesting experiment but it can’t be used as a per-value compressor at the moment. The resulting data IS transparent but it’s not quite so simple. We compress in blocks of 1024 and each block has a fixed size but also has some padding.

We do use this as a block compressor currently.

In other words, if we try the simple math to access the item at index i we will be out of luck because bits_per_value * i is not the location. What we need is something like:

let chunk_idx = i / 1024;
let chunk_offset = i % 1024;
bits_per_chunk * chunk_idx + bits_per_value * chunk_offset

However, this logic isn’t expressible with the per-value traits we have today. We can enhance these traits should we need to support it at some point in the future.

Implementations

impl OutOfLineBitpacking

pub fn new(compressed_bit_width: u64, uncompressed_bit_width: u64) -> Self

Trait Implementations

impl BlockCompressor for OutOfLineBitpacking

fn compress(&self, data: DataBlock) -> Result<LanceBuffer>

Compress the data into a single buffer

impl BlockDecompressor for OutOfLineBitpacking

fn decompress(&self, data: LanceBuffer, num_values: u64) -> Result<DataBlock>

impl Debug for OutOfLineBitpacking

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.