Struct datafusion::datasource::file_format::parquet::ParquetFormat

pub struct ParquetFormat { /* private fields */ }

The Apache Parquet FileFormat implementation

Note it is recommended these are instead configured on the ConfigOptions associated with the SessionState instead of overridden on a format-basis

TODO: Deprecate and remove overrides https://github.com/apache/arrow-datafusion/issues/4349

Implementations

impl ParquetFormat

pub fn new() -> Self

Construct a new Format with no local overrides

pub fn with_enable_pruning(self, enable: Option<bool>) -> Self

Activate statistics based row group level pruning

• If None, defaults to value on config_options

pub fn enable_pruning(&self, config_options: &ConfigOptions) -> bool

Return true if pruning is enabled

pub fn with_metadata_size_hint(self, size_hint: Option<usize>) -> Self

Provide a hint to the size of the file metadata. If a hint is provided the reader will try and fetch the last size_hint bytes of the parquet file optimistically. Without a hint, two read are required. One read to fetch the 8-byte parquet footer and then another read to fetch the metadata length encoded in the footer.

• If None, defaults to value on config_options

pub fn metadata_size_hint( &self, config_options: &ConfigOptions ) -> Option<usize>

Return the metadata size hint if set

pub fn with_skip_metadata(self, skip_metadata: Option<bool>) -> Self

Tell the parquet reader to skip any metadata that may be in the file Schema. This can help avoid schema conflicts due to metadata.

• If None, defaults to value on config_options

pub fn skip_metadata(&self, config_options: &ConfigOptions) -> bool

Returns true if schema metadata will be cleared prior to schema merging.

Trait Implementations

impl Debug for ParquetFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ParquetFormat

fn default() -> ParquetFormat

Returns the “default value” for a type.

impl FileFormat for ParquetFormat

fn as_any(&self) -> &dyn Any

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn infer_schema<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, state: &'life1 SessionState, store: &'life2 Arc<dyn ObjectStore>, objects: &'life3 [ObjectMeta] ) -> Pin<Box<dyn Future<Output = Result<SchemaRef>> + Send + 'async_trait>>where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Infer the common schema of the provided objects. The objects will usually be analysed up to a given number of records or files (as specified in the format config) then give the estimated common schema. This might fail if the files have schemas that cannot be merged.

fn infer_stats<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 SessionState, store: &'life2 Arc<dyn ObjectStore>, table_schema: SchemaRef, object: &'life3 ObjectMeta ) -> Pin<Box<dyn Future<Output = Result<Statistics>> + Send + 'async_trait>>where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Infer the statistics for the provided object. The cost and accuracy of the estimated statistics might vary greatly between file formats.

fn create_physical_plan<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, state: &'life1 SessionState, conf: FileScanConfig, filters: Option<&'life2 Arc<dyn PhysicalExpr>> ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Take a list of files and convert it to the appropriate executor according to this file format.

fn create_writer_physical_plan<'life0, 'life1, 'async_trait>( &'life0 self, input: Arc<dyn ExecutionPlan>, _state: &'life1 SessionState, conf: FileSinkConfig, order_requirements: Option<Vec<PhysicalSortRequirement>> ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Take a list of files and the configuration to convert it to the appropriate writer executor according to this file format.

fn file_type(&self) -> FileType

Returns the FileType corresponding to this FileFormat