Struct datafusion::datasource::file_format::options::ParquetReadOptions

pub struct ParquetReadOptions<'a> {
    pub file_extension: &'a str,
    pub table_partition_cols: Vec<(String, DataType)>,
    pub parquet_pruning: Option<bool>,
    pub skip_metadata: Option<bool>,
}

Options that control the reading of Parquet files.

Note this structure is supplied when a datasource is created and can not not vary from statement to statement. For settings that can vary statement to statement see ConfigOptions.

Fields

file_extension: &'a str

File extension; only files with this extension are selected for data input. Defaults to “.parquet”.

table_partition_cols: Vec<(String, DataType)>

Partition Columns

parquet_pruning: Option<bool>

Should the parquet reader use the predicate to prune row groups? If None, uses value in SessionConfig

skip_metadata: Option<bool>

Should the parquet reader to skip any metadata that may be in the file Schema? This can help avoid schema conflicts due to metadata.

If None specified, uses value in SessionConfig

Implementations

impl<'a> ParquetReadOptions<'a>

pub fn parquet_pruning(self, parquet_pruning: bool) -> Self

Specify parquet_pruning

pub fn skip_metadata(self, skip_metadata: bool) -> Self

Tell the parquet reader to skip any metadata that may be in the file Schema. This can help avoid schema conflicts due to metadata. Defaults to true.

pub fn table_partition_cols( self, table_partition_cols: Vec<(String, DataType)> ) -> Self

Specify table_partition_cols for partition pruning

Trait Implementations

impl<'a> Clone for ParquetReadOptions<'a>

fn clone(&self) -> ParquetReadOptions<'a>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Default for ParquetReadOptions<'a>

fn default() -> Self

Returns the “default value” for a type.

impl ReadOptions<'_> for ParquetReadOptions<'_>

fn to_listing_options(&self, config: &SessionConfig) -> ListingOptions

Helper to convert these user facing options to ListingTable options

fn get_resolved_schema<'life0, 'life1, 'async_trait>( &'life0 self, config: &'life1 SessionConfig, state: SessionState, table_path: ListingTableUrl ) -> Pin<Box<dyn Future<Output = Result<SchemaRef>> + Send + 'async_trait>>where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Infer and resolve the schema from the files/sources provided.

fn _get_resolved_schema<'life0, 'async_trait>( &'a self, config: &'life0 SessionConfig, state: SessionState, table_path: ListingTableUrl, schema: Option<&'a Schema>, infinite: bool ) -> Pin<Box<dyn Future<Output = Result<SchemaRef>> + Send + 'async_trait>>where Self: Sync + 'async_trait, 'a: 'async_trait, 'life0: 'async_trait,

helper function to reduce repetitive code. Infers the schema from sources if not provided. Infinite data sources not supported through this function.