Struct TableParquetOptions

pub struct TableParquetOptions {
    pub global: ParquetOptions,
    pub column_specific_options: HashMap<String, ParquetColumnOptions>,
    pub key_value_metadata: HashMap<String, Option<String>>,
}

Options that control how Parquet files are read, including global options that apply to all columns and optional column-specific overrides

Closely tied to ParquetWriterOptions. Properties not included in TableParquetOptions may not be configurable at the external API (e.g. sorting_columns).

Fields

global: ParquetOptions

Global Parquet options that propagates to all columns.

column_specific_options: HashMap<String, ParquetColumnOptions>

Column specific options. Default usage is parquet.XX::column.

key_value_metadata: HashMap<String, Option<String>>

Additional file-level metadata to include. Inserted into the key_value_metadata for the written FileMetaData.

Multiple entries are permitted

OPTIONS (
   'format.metadata::key1' '',
   'format.metadata::key2' 'value',
   'format.metadata::key3' 'value has spaces',
   'format.metadata::key4' 'value has special chars :: :',
   'format.metadata::key_dupe' 'original will be overwritten',
   'format.metadata::key_dupe' 'final'
)

Implementations

impl TableParquetOptions

pub fn new() -> Self

Return new default TableParquetOptions

pub fn with_skip_arrow_metadata(self, skip: bool) -> Self

Set whether the encoding of the arrow metadata should occur during the writing of parquet.

Default is to encode the arrow schema in the file kv_metadata.

impl TableParquetOptions

pub fn arrow_schema(&mut self, schema: &Arc<Schema>)

Available on crate feature parquet only.

Add the arrow schema to the parquet kv_metadata. If already exists, then overwrites.

Trait Implementations

impl Clone for TableParquetOptions

fn clone(&self) -> TableParquetOptions

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ConfigField for TableParquetOptions

fn visit<V: Visit>( &self, v: &mut V, key_prefix: &str, description: &'static str, )

fn set(&mut self, key: &str, value: &str) -> Result<()>

impl Debug for TableParquetOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TableParquetOptions

fn default() -> TableParquetOptions

Returns the “default value” for a type.

impl PartialEq for TableParquetOptions

fn eq(&self, other: &TableParquetOptions) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<&TableParquetOptions> for ParquetWriterOptions

Available on crate feature parquet only.

type Error = DataFusionError

The type returned in the event of a conversion error.

fn try_from(parquet_table_options: &TableParquetOptions) -> Result<Self>

Performs the conversion.

impl TryFrom<&TableParquetOptions> for WriterPropertiesBuilder

Available on crate feature parquet only.

fn try_from(table_parquet_options: &TableParquetOptions) -> Result<Self>

Convert the session’s TableParquetOptions into a single write action’s WriterPropertiesBuilder.

The returned WriterPropertiesBuilder includes customizations applicable per column.

type Error = DataFusionError

The type returned in the event of a conversion error.

impl StructuralPartialEq for TableParquetOptions