Struct OrcSerDe 

#[non_exhaustive]
pub struct OrcSerDe {
    pub stripe_size_bytes: Option<i32>,
    pub block_size_bytes: Option<i32>,
    pub row_index_stride: Option<i32>,
    pub enable_padding: Option<bool>,
    pub padding_tolerance: Option<f64>,
    pub compression: Option<OrcCompression>,
    pub bloom_filter_columns: Option<Vec<String>>,
    pub bloom_filter_false_positive_probability: Option<f64>,
    pub dictionary_key_threshold: Option<f64>,
    pub format_version: Option<OrcFormatVersion>,
}

A serializer to use for converting data to the ORC format before storing it in Amazon S3. For more information, see Apache ORC.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

stripe_size_bytes: Option<i32>

The number of bytes in each stripe. The default is 64 MiB and the minimum is 8 MiB.

block_size_bytes: Option<i32>

The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Firehose uses this value for padding calculations.

row_index_stride: Option<i32>

The number of rows between index entries. The default is 10,000 and the minimum is 1,000.

enable_padding: Option<bool>

Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is false.

padding_tolerance: Option<f64>

A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Firehose ignores this parameter when OrcSerDe$EnablePadding is false.

compression: Option<OrcCompression>

The compression code to use over data blocks. The default is SNAPPY.

bloom_filter_columns: Option<Vec<String>>

The column names for which you want Firehose to create bloom filters. The default is null.

bloom_filter_false_positive_probability: Option<f64>

The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.

dictionary_key_threshold: Option<f64>

Represents the fraction of the total number of non-null rows. To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.

format_version: Option<OrcFormatVersion>

The version of the file to write. The possible values are V0_11 and V0_12. The default is V0_12.

Implementations

impl OrcSerDe

pub fn stripe_size_bytes(&self) -> Option<i32>

The number of bytes in each stripe. The default is 64 MiB and the minimum is 8 MiB.

pub fn block_size_bytes(&self) -> Option<i32>

The Hadoop Distributed File System (HDFS) block size. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Firehose uses this value for padding calculations.

pub fn row_index_stride(&self) -> Option<i32>

The number of rows between index entries. The default is 10,000 and the minimum is 1,000.

pub fn enable_padding(&self) -> Option<bool>

Set this to true to indicate that you want stripes to be padded to the HDFS block boundaries. This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is false.

pub fn padding_tolerance(&self) -> Option<f64>

A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. The default value is 0.05, which means 5 percent of stripe size.

For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.

Firehose ignores this parameter when OrcSerDe$EnablePadding is false.

pub fn compression(&self) -> Option<&OrcCompression>

The compression code to use over data blocks. The default is SNAPPY.

pub fn bloom_filter_columns(&self) -> &[String]

The column names for which you want Firehose to create bloom filters. The default is null.

If no value was sent for this field, a default will be set. If you want to determine if no value was sent, use .bloom_filter_columns.is_none().

pub fn bloom_filter_false_positive_probability(&self) -> Option<f64>

The Bloom filter false positive probability (FPP). The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.

pub fn dictionary_key_threshold(&self) -> Option<f64>

Represents the fraction of the total number of non-null rows. To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.

pub fn format_version(&self) -> Option<&OrcFormatVersion>

The version of the file to write. The possible values are V0_11 and V0_12. The default is V0_12.

impl OrcSerDe

pub fn builder() -> OrcSerDeBuilder

Creates a new builder-style object to manufacture OrcSerDe.

Trait Implementations

impl Clone for OrcSerDe

fn clone(&self) -> OrcSerDe

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for OrcSerDe

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for OrcSerDe

fn eq(&self, other: &OrcSerDe) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for OrcSerDe