Trait lance_encoding::encoder::ArrayEncoder

source · 
pub trait ArrayEncoder: Debug + Send + Sync {
    // Required method
    fn encode(
        &self,
        arrays: &[ArrayRef],
        buffer_index: &mut u32,
    ) -> Result<EncodedArray>;
}
Expand description

Encodes data from Arrow format into some kind of on-disk format

The encoder is responsible for looking at the incoming data and determining which encoding is most appropriate. This may involve calculating statistics, etc. It then needs to actually encode that data according to the chosen encoding.

The encoder may even encode the statistics as well (typically in the column metadata) so that the statistics can be used for filtering later.

The array encoder must be Send + Sync. Encoding is always done on its own thread task in the background and there could potentially be multiple encode tasks running for a column at once.

Note: not all Arrow arrays can be encoded using an ArrayEncoder. Some arrays will be econded into several Lance columns. For example, a list array or a struct array. See FieldEncoder for the top-level encoding entry point

Required Methods§

source

fn encode( &self, arrays: &[ArrayRef], buffer_index: &mut u32, ) -> Result<EncodedArray>

Encode data

This method may receive multiple chunks and should encode them into a single EncodedPage.

The result should contain a description of the encoding that was chosen. This can be used to decode the data later.

Implementors§