Struct aws_sdk_glue::types::S3ParquetSource

#[non_exhaustive]
pub struct S3ParquetSource {
    pub name: String,
    pub paths: Vec<String>,
    pub compression_type: Option<ParquetCompressionType>,
    pub exclusions: Option<Vec<String>>,
    pub group_size: Option<String>,
    pub group_files: Option<String>,
    pub recurse: Option<bool>,
    pub max_band: Option<i32>,
    pub max_files_in_band: Option<i32>,
    pub additional_options: Option<S3DirectSourceAdditionalOptions>,
    pub output_schemas: Option<Vec<GlueSchema>>,
}

Specifies an Apache Parquet data store stored in Amazon S3.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

name: String

The name of the data store.

paths: Vec<String>

A list of the Amazon S3 paths to read from.

compression_type: Option<ParquetCompressionType>

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

exclusions: Option<Vec<String>>

A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

group_size: Option<String>

The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles" must be set to "inPartition" for this to take effect.

group_files: Option<String>

Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none".

recurse: Option<bool>

If set to true, recursively reads files in all subdirectories under the specified paths.

max_band: Option<i32>

This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.

max_files_in_band: Option<i32>

This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.

additional_options: Option<S3DirectSourceAdditionalOptions>

Specifies additional connection options.

output_schemas: Option<Vec<GlueSchema>>

Specifies the data schema for the S3 Parquet source.

Implementations

impl S3ParquetSource

pub fn name(&self) -> &str

The name of the data store.

pub fn paths(&self) -> &[String]

A list of the Amazon S3 paths to read from.

pub fn compression_type(&self) -> Option<&ParquetCompressionType>

Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip" and "bzip").

pub fn exclusions(&self) -> &[String]

A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.

If no value was sent for this field, a default will be set. If you want to determine if no value was sent, use .exclusions.is_none().

pub fn group_size(&self) -> Option<&str>

The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles" must be set to "inPartition" for this to take effect.

pub fn group_files(&self) -> Option<&str>

Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none".

pub fn recurse(&self) -> Option<bool>

If set to true, recursively reads files in all subdirectories under the specified paths.

pub fn max_band(&self) -> Option<i32>

This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.

pub fn max_files_in_band(&self) -> Option<i32>

This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.

pub fn additional_options(&self) -> Option<&S3DirectSourceAdditionalOptions>

Specifies additional connection options.

pub fn output_schemas(&self) -> &[GlueSchema]

Specifies the data schema for the S3 Parquet source.

If no value was sent for this field, a default will be set. If you want to determine if no value was sent, use .output_schemas.is_none().

impl S3ParquetSource

pub fn builder() -> S3ParquetSourceBuilder

Creates a new builder-style object to manufacture S3ParquetSource.

Trait Implementations

impl Clone for S3ParquetSource

fn clone(&self) -> S3ParquetSource

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for S3ParquetSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for S3ParquetSource

fn eq(&self, other: &S3ParquetSource) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for S3ParquetSource