Struct parquet::encoding::DictEncoder

pub struct DictEncoder<T: DataType> { /* fields omitted */ }

Dictionary encoder. The dictionary encoding builds a dictionary of values encountered in a given column. The dictionary page is written first, before the data pages of the column chunk.

Dictionary page format: the entries in the dictionary - in dictionary order - using the plain encoding.

Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32), followed by the values encoded using RLE/Bit packed described above (with the given bit width).

Implementations

impl<T: DataType> DictEncoder<T>

pub fn new(desc: ColumnDescPtr, mem_tracker: MemTrackerPtr) -> Self

Creates new dictionary encoder.

pub fn is_sorted(&self) -> bool

Returns true if dictionary entries are sorted, false otherwise.

pub fn num_entries(&self) -> usize

Returns number of unique values (keys) in the dictionary.

pub fn dict_encoded_size(&self) -> usize

Returns size of unique values (keys) in the dictionary, in bytes.

pub fn write_dict(&self) -> Result<ByteBufferPtr>

Writes out the dictionary values with PLAIN encoding in a byte buffer, and return the result.

pub fn write_indices(&mut self) -> Result<ByteBufferPtr>

Writes out the dictionary values with RLE encoding in a byte buffer, and return the result.

Trait Implementations

impl<T: DataType> Encoder<T> for DictEncoder<T>

fn put(&mut self, values: &[T::T]) -> Result<()>

Encodes data from values.

fn encoding(&self) -> Encoding

Returns the encoding type of this encoder.

fn estimated_data_encoded_size(&self) -> usize

Returns an estimate of the encoded data, in bytes. Method call must be O(1).

fn flush_buffer(&mut self) -> Result<ByteBufferPtr>

Flushes the underlying byte buffer that’s being processed by this encoder, and return the immutable copy of it. This will also reset the internal state.

fn put_spaced(&mut self, values: &[T::T], valid_bits: &[u8]) -> Result<usize>

Encodes data from values, which contains spaces for null values, that is identified by valid_bits.