Struct TableParquetOptions 

pub struct TableParquetOptions {
    pub global: ParquetOptions,
    pub column_specific_options: HashMap<String, ParquetColumnOptions>,
    pub key_value_metadata: HashMap<String, Option<String>>,
    pub crypto: ParquetEncryptionOptions,
}

Options that control how Parquet files are read, including global options that apply to all columns and optional column-specific overrides

Closely tied to ParquetWriterOptions (see crate::file_options::parquet_writer::ParquetWriterOptions when the “parquet” feature is enabled). Properties not included in TableParquetOptions may not be configurable at the external API (e.g. sorting_columns).

Fields

global: ParquetOptions

Global Parquet options that propagates to all columns.

column_specific_options: HashMap<String, ParquetColumnOptions>

Column specific options. Default usage is parquet.XX::column.

key_value_metadata: HashMap<String, Option<String>>

Additional file-level metadata to include. Inserted into the key_value_metadata for the written FileMetaData.

Multiple entries are permitted

OPTIONS (
   'format.metadata::key1' '',
   'format.metadata::key2' 'value',
   'format.metadata::key3' 'value has spaces',
   'format.metadata::key4' 'value has special chars :: :',
   'format.metadata::key_dupe' 'original will be overwritten',
   'format.metadata::key_dupe' 'final'
)

crypto: ParquetEncryptionOptions

Options for configuring Parquet modular encryption

To use Parquet encryption, you must enable the parquet_encryption feature flag, as it is not activated by default. See ConfigFileEncryptionProperties and ConfigFileDecryptionProperties in datafusion/common/src/config.rs These can be set via ‘format.crypto’, for example:

OPTIONS (
   'format.crypto.file_encryption.encrypt_footer' 'true',
   'format.crypto.file_encryption.footer_key_as_hex' '30313233343536373839303132333435',  -- b"0123456789012345" */
   'format.crypto.file_encryption.column_key_as_hex::double_field' '31323334353637383930313233343530', -- b"1234567890123450"
   'format.crypto.file_encryption.column_key_as_hex::float_field' '31323334353637383930313233343531', -- b"1234567890123451"
    -- Same for decryption
   'format.crypto.file_decryption.footer_key_as_hex' '30313233343536373839303132333435', -- b"0123456789012345"
   'format.crypto.file_decryption.column_key_as_hex::double_field' '31323334353637383930313233343530', -- b"1234567890123450"
   'format.crypto.file_decryption.column_key_as_hex::float_field' '31323334353637383930313233343531', -- b"1234567890123451"
)

See datafusion-cli/tests/sql/encrypted_parquet.sql for a more complete example. Note that keys must be provided as in hex format since these are binary strings.

Implementations

impl TableParquetOptions

pub fn new() -> Self

Return new default TableParquetOptions

pub fn with_skip_arrow_metadata(self, skip: bool) -> Self

Set whether the encoding of the arrow metadata should occur during the writing of parquet.

Default is to encode the arrow schema in the file kv_metadata.

pub fn entries(self: &TableParquetOptions) -> Vec<ConfigEntry>

Retrieves all configuration entries from this TableParquetOptions.

Returns

A vector of ConfigEntry instances, representing all the configuration options within this

impl TableParquetOptions

pub fn arrow_schema(&mut self, schema: &Arc<Schema>)

Available on crate feature parquet only.

Add the arrow schema to the parquet kv_metadata. If already exists, then overwrites.

Trait Implementations

impl Clone for TableParquetOptions

fn clone(&self) -> TableParquetOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ConfigField for TableParquetOptions

fn visit<V: Visit>( &self, v: &mut V, key_prefix: &str, description: &'static str, )

fn set(&mut self, key: &str, value: &str) -> Result<()>

fn reset(&mut self, key: &str) -> Result<()>

impl Debug for TableParquetOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TableParquetOptions

fn default() -> TableParquetOptions

Returns the “default value” for a type.

impl PartialEq for TableParquetOptions

fn eq(&self, other: &TableParquetOptions) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TableParquetOptions

impl TryFrom<&TableParquetOptions> for ParquetWriterOptions

Available on crate feature parquet only.

type Error = DataFusionError

The type returned in the event of a conversion error.

fn try_from(parquet_table_options: &TableParquetOptions) -> Result<Self>

Performs the conversion.

impl TryFrom<&TableParquetOptions> for WriterPropertiesBuilder

Available on crate feature parquet only.

fn try_from(table_parquet_options: &TableParquetOptions) -> Result<Self>

Convert the session’s TableParquetOptions into a single write action’s WriterPropertiesBuilder.

The returned WriterPropertiesBuilder includes customizations applicable per column. Note that any encryption options are ignored as building the FileEncryptionProperties might require other inputs besides the TableParquetOptions.

type Error = DataFusionError

The type returned in the event of a conversion error.