Struct ParquetCdcOptions 

pub struct ParquetCdcOptions {
    pub enabled: bool,
    pub min_chunk_size: usize,
    pub max_chunk_size: usize,
    pub norm_level: i32,
}

Options for content-defined chunking (CDC) when writing parquet files. Mirrors parquet::file::properties::CdcOptions.

Carried as a ParquetCdcOptions in ParquetOptions::content_defined_chunking with an explicit enabled flag, so it can be toggled with dotted config keys (content_defined_chunking.enabled = true|false) and the result is independent of the order in which the keys are set.

Fields

enabled: bool

(writing) EXPERIMENTAL: Enable content-defined chunking (CDC) when writing parquet files. When enabled, parallel writing is automatically disabled since the chunker state must persist across row groups.

min_chunk_size: usize

Minimum chunk size in bytes. The rolling hash will not trigger a split until this many bytes have been accumulated. Default is 256 KiB.

max_chunk_size: usize

Maximum chunk size in bytes. A split is forced when the accumulated size exceeds this value. Default is 1 MiB.

norm_level: i32

Normalization level. Increasing this improves deduplication ratio but increases fragmentation. Recommended range is [-3, 3], default is 0.

Implementations

impl ParquetCdcOptions

pub fn enabled() -> ParquetCdcOptions

Returns enabled CDC options with the default chunking parameters.

Shorthand for ParquetCdcOptions { enabled: true, ..Default::default() }; combine with struct-update syntax to override parameters, e.g. ParquetCdcOptions { min_chunk_size: 4096, ..ParquetCdcOptions::enabled() }.

pub fn disabled() -> ParquetCdcOptions

Returns disabled CDC options (equivalent to ParquetCdcOptions::default).

Trait Implementations

impl Clone for ParquetCdcOptions

fn clone(&self) -> ParquetCdcOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ConfigField for ParquetCdcOptions

fn set(&mut self, key: &str, value: &str) -> Result<(), DataFusionError>

fn visit<V>(&self, v: &mut V, key_prefix: &str, _description: &'static str)

where V: Visit,

fn reset(&mut self, key: &str) -> Result<(), DataFusionError>

impl Debug for ParquetCdcOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for ParquetCdcOptions

fn default() -> ParquetCdcOptions

Returns the “default value” for a type.

impl From<&ParquetCdcOptions> for Option<CdcOptions>

Convert DataFusion’s ParquetCdcOptions into parquet-rs’s Option<CdcOptions>.

parquet-rs has no enabled flag; CDC is on when the option is Some. So a disabled ParquetCdcOptions maps to None, and an enabled one to Some with the chunking parameters.

fn from(value: &ParquetCdcOptions) -> Option<CdcOptions>

Converts to this type from the input type.

impl From<Option<&CdcOptions>> for ParquetCdcOptions

Convert parquet-rs’s Option<&CdcOptions> back into DataFusion’s ParquetCdcOptions.

The presence of parquet-rs options means CDC was enabled, so Some maps to enabled: true; None yields the disabled default.

fn from(value: Option<&CdcOptions>) -> ParquetCdcOptions

Converts to this type from the input type.

impl From<ParquetCdcOptions> for ParquetCdcOptions

fn from(value: ParquetCdcOptions) -> ParquetCdcOptions

Converts to this type from the input type.

impl PartialEq for ParquetCdcOptions

fn eq(&self, other: &ParquetCdcOptions) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for ParquetCdcOptions