Struct TableParquetOptions 

pub struct TableParquetOptions {
    pub global: ParquetOptions,
    pub column_specific_options: HashMap<String, ParquetColumnOptions>,
    pub key_value_metadata: HashMap<String, Option<String>>,
    pub crypto: ParquetEncryptionOptions,
}

Options that control how Parquet files are read, including global options that apply to all columns and optional column-specific overrides

Closely tied to ParquetWriterOptions. Properties not included in TableParquetOptions may not be configurable at the external API (e.g. sorting_columns).

Fields

global: ParquetOptions

Global Parquet options that propagates to all columns.

column_specific_options: HashMap<String, ParquetColumnOptions>

Column specific options. Default usage is parquet.XX::column.

key_value_metadata: HashMap<String, Option<String>>

Additional file-level metadata to include. Inserted into the key_value_metadata for the written FileMetaData.

Multiple entries are permitted

OPTIONS (
   'format.metadata::key1' '',
   'format.metadata::key2' 'value',
   'format.metadata::key3' 'value has spaces',
   'format.metadata::key4' 'value has special chars :: :',
   'format.metadata::key_dupe' 'original will be overwritten',
   'format.metadata::key_dupe' 'final'
)

crypto: ParquetEncryptionOptions

Options for configuring Parquet modular encryption

To use Parquet encryption, you must enable the parquet_encryption feature flag, as it is not activated by default. See ConfigFileEncryptionProperties and ConfigFileDecryptionProperties in datafusion/common/src/config.rs These can be set via ‘format.crypto’, for example:

OPTIONS (
   'format.crypto.file_encryption.encrypt_footer' 'true',
   'format.crypto.file_encryption.footer_key_as_hex' '30313233343536373839303132333435',  -- b"0123456789012345" */
   'format.crypto.file_encryption.column_key_as_hex::double_field' '31323334353637383930313233343530', -- b"1234567890123450"
   'format.crypto.file_encryption.column_key_as_hex::float_field' '31323334353637383930313233343531', -- b"1234567890123451"
    -- Same for decryption
   'format.crypto.file_decryption.footer_key_as_hex' '30313233343536373839303132333435', -- b"0123456789012345"
   'format.crypto.file_decryption.column_key_as_hex::double_field' '31323334353637383930313233343530', -- b"1234567890123450"
   'format.crypto.file_decryption.column_key_as_hex::float_field' '31323334353637383930313233343531', -- b"1234567890123451"
)

See datafusion-cli/tests/sql/encrypted_parquet.sql for a more complete example. Note that keys must be provided as in hex format since these are binary strings.

Implementations

impl TableParquetOptions

pub fn new() -> TableParquetOptions

Return new default TableParquetOptions

pub fn with_skip_arrow_metadata(self, skip: bool) -> TableParquetOptions

Set whether the encoding of the arrow metadata should occur during the writing of parquet.

Default is to encode the arrow schema in the file kv_metadata.

pub fn entries(&self) -> Vec<ConfigEntry>

Retrieves all configuration entries from this TableParquetOptions.

Returns

A vector of ConfigEntry instances, representing all the configuration options within this

impl TableParquetOptions

pub fn arrow_schema(&mut self, schema: &Arc<Schema>)

Add the arrow schema to the parquet kv_metadata. If already exists, then overwrites.

Trait Implementations

impl Clone for TableParquetOptions

fn clone(&self) -> TableParquetOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ConfigField for TableParquetOptions

fn visit<V>(&self, v: &mut V, key_prefix: &str, description: &'static str)

where V: Visit,

fn set(&mut self, key: &str, value: &str) -> Result<(), DataFusionError>

impl Debug for TableParquetOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for TableParquetOptions

fn default() -> TableParquetOptions

Returns the “default value” for a type.

impl From<&TableParquetOptions> for TableParquetOptions

fn from(proto: &TableParquetOptions) -> TableParquetOptions

Converts to this type from the input type.

impl PartialEq for TableParquetOptions

fn eq(&self, other: &TableParquetOptions) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<&TableParquetOptions> for ParquetWriterOptions

type Error = DataFusionError

The type returned in the event of a conversion error.

fn try_from( parquet_table_options: &TableParquetOptions, ) -> Result<ParquetWriterOptions, DataFusionError>

Performs the conversion.

impl TryFrom<&TableParquetOptions> for TableParquetOptions

type Error = DataFusionError

The type returned in the event of a conversion error.

fn try_from( value: &TableParquetOptions, ) -> Result<TableParquetOptions, <TableParquetOptions as TryFrom<&TableParquetOptions>>::Error>

Performs the conversion.

impl StructuralPartialEq for TableParquetOptions