Struct PartitionColumnProjector

pub struct PartitionColumnProjector { /* private fields */ }

A helper that projects partition columns into the file record batches.

One interesting trick is the usage of a cache for the key buffers of the partition column dictionaries. Indeed, the partition columns are constant, so the dictionaries that represent them have all their keys equal to 0. This enables us to re-use the same “all-zero” buffer across batches, which makes the space consumption of the partition columns O(batch_size) instead of O(record_count).

Implementations

impl PartitionColumnProjector

pub fn new(projected_schema: SchemaRef, table_partition_cols: &[String]) -> Self

pub fn project( &mut self, file_batch: RecordBatch, partition_values: &[ScalarValue], ) -> Result<RecordBatch>