Struct DefaultParquetHandler 

pub struct DefaultParquetHandler<E: TaskExecutor> { /* private fields */ }

Implementations

impl<E: TaskExecutor> DefaultParquetHandler<E>

pub fn new(store: Arc<DynObjectStore>, task_executor: Arc<E>) -> Self

pub fn with_buffer_size(self, buffer_size: NonZero<usize>) -> Self

Set the maximum number of files to read concurrently in Self::read_parquet_files().

Defaults to super::DEFAULT_READ_BUFFER_SIZE.

This setting applies only to object-store reads. When every file in the batch uses a presigned URL (https://...), reads bypass the object store and this value has no effect; use Self::with_batch_size to tune RecordBatch chunking in that path.

Memory constraints can be imposed by constraining the buffer size and batch size. Note that overall memory usage is proportional to the product of these two values.

1. Batch size governs the size of RecordBatches yielded in each iteration of the stream.
2. Buffer size governs the number of concurrent file reads (which equals the size of the readahead buffer).

pub fn with_batch_size(self, batch_size: NonZero<usize>) -> Self

Set the maximum number of rows per RecordBatch yielded by Self::read_parquet_files().

Defaults to super::DEFAULT_READ_BATCH_SIZE rows.

pub async fn write_parquet_file( &self, data: Box<dyn EngineData>, write_context: &BoundWriteContext, ) -> KernelResult<Box<dyn EngineData>>

Write data to a new parquet file under the BoundWriteContext::write_dir and return Add action metadata ready for Transaction::add_files.

The metadata schema does not contain the dataChange column. Kernel uses the data_change flag supplied when constructing the transaction builder.

Trait Implementations

impl<E: Debug + TaskExecutor> Debug for DefaultParquetHandler<E>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<E: TaskExecutor> ParquetHandler for DefaultParquetHandler<E>

fn write_parquet_file( &self, location: Url, data: KernelResultIteratorStatic<Box<dyn EngineData>>, ) -> KernelResult<FileSize>

Writes engine data to a Parquet file at the specified location.

This implementation uses asynchronous file I/O with object_store to write the Parquet file. If a file already exists at the given location, it will be overwritten.

Parameters

• location - The full URL path where the Parquet file should be written (e.g., s3://bucket/path/file.parquet, file:///path/to/file.parquet).
• data - An iterator of engine data to be written to the Parquet file.

Returns

The exact number of bytes written to the parquet file.

fn read_parquet_files( &self, files: &[FileMeta], physical_schema: SchemaRef, predicate: Option<PredicateRef>, ) -> KernelResult<FileDataReadResultIterator>

Read and parse the Parquet file at given locations and return the data as EngineData with the columns requested by physical schema. The ParquetHandler must return exactly the columns specified in physical_schema, and they must be in schema order.

fn read_parquet_files_with_cancellation( &self, files: &[FileMeta], physical_schema: SchemaRef, predicate: Option<PredicateRef>, cancellation_token: Option<CancellationTokenRef>, ) -> KernelResult<FileDataReadResultIterator>

Cancellation-aware variant of read_parquet_files.

fn read_parquet_footer(&self, file: &FileMeta) -> KernelResult<ParquetFooter>

Read the footer metadata from a Parquet file without reading the data.

fn read_parquet_footer_with_cancellation( &self, file: &FileMeta, cancellation_token: Option<CancellationTokenRef>, ) -> KernelResult<ParquetFooter>

Cancellation-aware variant of read_parquet_footer.