Struct ParquetFormat 

pub struct ParquetFormat { /* private fields */ }

The Apache Parquet FileFormat implementation

Implementations

impl ParquetFormat

pub fn new() -> Self

Construct a new Format with no local overrides

pub fn with_enable_pruning(self, enable: bool) -> Self

Activate statistics based row group level pruning

• If None, defaults to value on config_options

pub fn enable_pruning(&self) -> bool

Return true if pruning is enabled

pub fn with_metadata_size_hint(self, size_hint: Option<usize>) -> Self

Provide a hint to the size of the file metadata. If a hint is provided the reader will try and fetch the last size_hint bytes of the parquet file optimistically. Without a hint, two read are required. One read to fetch the 8-byte parquet footer and then another read to fetch the metadata length encoded in the footer.

• If None, defaults to value on config_options

pub fn metadata_size_hint(&self) -> Option<usize>

Return the metadata size hint if set

pub fn with_skip_metadata(self, skip_metadata: bool) -> Self

Tell the parquet reader to skip any metadata that may be in the file Schema. This can help avoid schema conflicts due to metadata.

• If None, defaults to value on config_options

pub fn skip_metadata(&self) -> bool

Returns true if schema metadata will be cleared prior to schema merging.

pub fn with_options(self, options: TableParquetOptions) -> Self

Set Parquet options for the ParquetFormat

pub fn options(&self) -> &TableParquetOptions

Parquet options

pub fn force_view_types(&self) -> bool

Return true if should use view types.

If this returns true, DataFusion will instruct the parquet reader to read string / binary columns using view StringView or BinaryView if the table schema specifies those types, regardless of any embedded metadata that may specify an alternate Arrow type. The parquet reader is optimized for reading StringView and BinaryView and such queries are significantly faster.

If this returns false, the parquet reader will read the columns according to the defaults or any embedded Arrow type information. This may result in reading StringArrays and then casting to StringViewArray which is less efficient.

pub fn with_force_view_types(self, use_views: bool) -> Self

If true, will use view types. See Self::force_view_types for details

pub fn binary_as_string(&self) -> bool

Return true if binary types will be read as strings.

If this returns true, DataFusion will instruct the parquet reader to read binary columns such as Binary or BinaryView as the corresponding string type such as Utf8 or LargeUtf8. The parquet reader has special optimizations for Utf8 and LargeUtf8 validation, and such queries are significantly faster than reading binary columns and then casting to string columns.

pub fn with_binary_as_string(self, binary_as_string: bool) -> Self

If true, will read binary types as strings. See Self::binary_as_string for details

pub fn coerce_int96(&self) -> Option<String>

pub fn with_coerce_int96(self, time_unit: Option<String>) -> Self

Trait Implementations

impl Debug for ParquetFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ParquetFormat

fn default() -> ParquetFormat

Returns the “default value” for a type.

impl FileFormat for ParquetFormat

fn as_any(&self) -> &dyn Any

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn get_ext(&self) -> String

Returns the extension for this FileFormat, e.g. “file.csv” -> csv

fn get_ext_with_compression( &self, file_compression_type: &FileCompressionType, ) -> Result<String>

Returns the extension for this FileFormat when compressed, e.g. “file.csv.gz” -> csv

fn compression_type(&self) -> Option<FileCompressionType>

Returns whether this instance uses compression if applicable

fn infer_schema<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, state: &'life1 dyn Session, store: &'life2 Arc<dyn ObjectStore>, objects: &'life3 [ObjectMeta], ) -> Pin<Box<dyn Future<Output = Result<SchemaRef>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Infer the common schema of the provided objects. The objects will usually be analysed up to a given number of records or files (as specified in the format config) then give the estimated common schema. This might fail if the files have schemas that cannot be merged.

fn infer_stats<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, store: &'life2 Arc<dyn ObjectStore>, table_schema: SchemaRef, object: &'life3 ObjectMeta, ) -> Pin<Box<dyn Future<Output = Result<Statistics>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Infer the statistics for the provided object. The cost and accuracy of the estimated statistics might vary greatly between file formats.

fn create_physical_plan<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, conf: FileScanConfig, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Take a list of files and convert it to the appropriate executor according to this file format.

fn create_writer_physical_plan<'life0, 'life1, 'async_trait>( &'life0 self, input: Arc<dyn ExecutionPlan>, _state: &'life1 dyn Session, conf: FileSinkConfig, order_requirements: Option<LexRequirement>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Take a list of files and the configuration to convert it to the appropriate writer executor according to this file format.

fn file_source(&self) -> Arc<dyn FileSource>

Return the related FileSource such as CsvSource, JsonSource, etc.