Trait lance_encoding::decoder::PhysicalPageDecoder

pub trait PhysicalPageDecoder: Send + Sync {
    // Required methods
    fn update_capacity(
        &self,
        rows_to_skip: u32,
        num_rows: u32,
        buffers: &mut [(u64, bool)],
        all_null: &mut bool
    );
    fn decode_into(
        &self,
        rows_to_skip: u32,
        num_rows: u32,
        dest_buffers: &mut [BytesMut]
    ) -> Result<()>;
    fn num_buffers(&self) -> u32;
}

A decoder for single-column encodings of primitive data (this includes fixed size lists of primitive data)

Physical decoders are able to decode into existing buffers for zero-copy operation.

Instances should be stateless and Send / Sync. This is because multiple decode tasks could reference the same page. For example, imagine a page covers rows 0-2000 and the decoder stream has a batch size of 1024. The decoder will be needed by both the decode task for batch 0 and the decode task for batch 1.

See crate::decoder for more information

Required Methods

fn update_capacity( &self, rows_to_skip: u32, num_rows: u32, buffers: &mut [(u64, bool)], all_null: &mut bool )

Calculates and updates the capacity required to represent the requested data

Capacity is stored as a tuple of (num_bytes: u64, is_needed: bool). The is_needed portion only needs to be updated if the encoding has some concept of an “optional” buffer.

The decoder should look at rows_to_skip and num_rows and then calculate how many bytes of data are needed. It should then update the first part of the tuple.

Note: Most encodings deal with a single buffer. They may have multiple input buffers but they only have a single output buffer. The current exception to this rule is the basic encoding which has an output “validity” buffer and an output “values” buffers. We may find there are other such exceptions.

Arguments

• rows_to_skip - how many rows to skip (within the page) before decoding
• num_rows - how many rows to decode
• buffers - A mutable slice of “capacities” (as described above), one per buffer
• all_null - A mutable bool, set to true if a decoder determines all values are null

fn decode_into( &self, rows_to_skip: u32, num_rows: u32, dest_buffers: &mut [BytesMut] ) -> Result<()>

Decodes the data into the requested buffers.

You can assume that the capacity will have already been configured on the BytesMut according to the capacity calculated in PhysicalPageDecoder::update_capacity

Arguments

• rows_to_skip - how many rows to skip (within the page) before decoding
• num_rows - how many rows to decode
• dest_buffers - the output buffers to decode into

fn num_buffers(&self) -> u32

Implementors

impl PhysicalPageDecoder for FixedListDecoder