Struct deltalake::datafusion::datasource::file_format::options::ParquetReadOptions

pub struct ParquetReadOptions<'a> {
    pub file_extension: &'a str,
    pub table_partition_cols: Vec<(String, DataType)>,
    pub parquet_pruning: Option<bool>,
    pub skip_metadata: Option<bool>,
    pub schema: Option<&'a Schema>,
    pub file_sort_order: Vec<Vec<Expr>>,
}

Options that control the reading of Parquet files.

Note this structure is supplied when a datasource is created and can not not vary from statement to statement. For settings that can vary statement to statement see ConfigOptions.

Fields

file_extension: &'a str

File extension; only files with this extension are selected for data input. Defaults to “.parquet”.

table_partition_cols: Vec<(String, DataType)>

Partition Columns

parquet_pruning: Option<bool>

Should the parquet reader use the predicate to prune row groups? If None, uses value in SessionConfig

skip_metadata: Option<bool>

Should the parquet reader to skip any metadata that may be in the file Schema? This can help avoid schema conflicts due to metadata.

If None specified, uses value in SessionConfig

schema: Option<&'a Schema>

An optional schema representing the parquet files. If None, parquet reader will try to infer it based on data in file.

file_sort_order: Vec<Vec<Expr>>

Indicates how the file is sorted

Implementations

impl<'a> ParquetReadOptions<'a>

pub fn parquet_pruning(self, parquet_pruning: bool) -> ParquetReadOptions<'a>

Specify parquet_pruning

pub fn skip_metadata(self, skip_metadata: bool) -> ParquetReadOptions<'a>

Tell the parquet reader to skip any metadata that may be in the file Schema. This can help avoid schema conflicts due to metadata. Defaults to true.

pub fn schema(self, schema: &'a Schema) -> ParquetReadOptions<'a>

Specify schema to use for parquet read

pub fn table_partition_cols( self, table_partition_cols: Vec<(String, DataType)> ) -> ParquetReadOptions<'a>

Specify table_partition_cols for partition pruning

pub fn file_sort_order( self, file_sort_order: Vec<Vec<Expr>> ) -> ParquetReadOptions<'a>

Configure if file has known sort order

Trait Implementations

impl<'a> Clone for ParquetReadOptions<'a>

fn clone(&self) -> ParquetReadOptions<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Default for ParquetReadOptions<'a>

fn default() -> ParquetReadOptions<'a>

Returns the “default value” for a type.

impl ReadOptions<'_> for ParquetReadOptions<'_>

fn to_listing_options( &self, config: &SessionConfig, table_options: TableOptions ) -> ListingOptions

Helper to convert these user facing options to ListingTable options

fn get_resolved_schema<'life0, 'life1, 'async_trait>( &'life0 self, config: &'life1 SessionConfig, state: SessionState, table_path: ListingTableUrl ) -> Pin<Box<dyn Future<Output = Result<Arc<Schema>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, ParquetReadOptions<'_>: 'async_trait,

Infer and resolve the schema from the files/sources provided.

fn _get_resolved_schema<'life0, 'async_trait>( &'a self, config: &'life0 SessionConfig, state: SessionState, table_path: ListingTableUrl, schema: Option<&'a Schema> ) -> Pin<Box<dyn Future<Output = Result<Arc<Schema>, DataFusionError>> + Send + 'async_trait>>

where 'a: 'async_trait, 'a: 'async_trait, 'life0: 'async_trait, Self: Sync + 'async_trait,

helper function to reduce repetitive code. Infers the schema from sources if not provided. Infinite data sources not supported through this function.