Struct deltalake::datafusion::parquet::file::properties::BloomFilterProperties

pub struct BloomFilterProperties {
    pub fpp: f64,
    pub ndv: u64,
}

Controls the bloom filter to be computed by the writer.

Fields

fpp: f64

False positive probability, should be always between 0 and 1 exclusive. Defaults to DEFAULT_BLOOM_FILTER_FPP.

You should set this value by calling WriterPropertiesBuilder::set_bloom_filter_fpp.

The bloom filter data structure is a trade of between disk and memory space versus fpp, the smaller the fpp, the more memory and disk space is required, thus setting it to a reasonable value e.g. 0.1, 0.05, or 0.001 is recommended.

Setting to very small number diminishes the value of the filter itself, as the bitset size is even larger than just storing the whole value. You are also expected to set ndv if it can be known in advance in order to largely reduce space usage.

ndv: u64

Number of distinct values, should be non-negative to be meaningful. Defaults to DEFAULT_BLOOM_FILTER_NDV.

You should set this value by calling WriterPropertiesBuilder::set_bloom_filter_ndv.

Usage of bloom filter is most beneficial for columns with large cardinality, so a good heuristic is to set ndv to number of rows. However it can reduce disk size if you know in advance a smaller number of distinct values. For very small ndv value it is probably not worth it to use bloom filter anyway.

Increasing this value (without increasing fpp) will result in an increase in disk or memory size.

Trait Implementations

impl Clone for BloomFilterProperties

fn clone(&self) -> BloomFilterProperties

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BloomFilterProperties

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for BloomFilterProperties

fn default() -> BloomFilterProperties

Returns the “default value” for a type.

impl PartialEq for BloomFilterProperties

fn eq(&self, other: &BloomFilterProperties) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for BloomFilterProperties