Struct aws_sdk_glue::types::S3JsonSource
source · #[non_exhaustive]pub struct S3JsonSource {Show 13 fields
pub name: String,
pub paths: Vec<String>,
pub compression_type: Option<CompressionType>,
pub exclusions: Option<Vec<String>>,
pub group_size: Option<String>,
pub group_files: Option<String>,
pub recurse: Option<bool>,
pub max_band: Option<i32>,
pub max_files_in_band: Option<i32>,
pub additional_options: Option<S3DirectSourceAdditionalOptions>,
pub json_path: Option<String>,
pub multiline: Option<bool>,
pub output_schemas: Option<Vec<GlueSchema>>,
}
Expand description
Specifies a JSON data store stored in Amazon S3.
Fields (Non-exhaustive)§
This struct is marked as non-exhaustive
Struct { .. }
syntax; cannot be matched against without a wildcard ..
; and struct update syntax will not work.name: String
The name of the data store.
paths: Vec<String>
A list of the Amazon S3 paths to read from.
compression_type: Option<CompressionType>
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip"
and "bzip"
).
exclusions: Option<Vec<String>>
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "\[\"**.pdf\"\]" excludes all PDF files.
group_size: Option<String>
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles"
must be set to "inPartition"
for this to take effect.
group_files: Option<String>
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none"
.
recurse: Option<bool>
If set to true, recursively reads files in all subdirectories under the specified paths.
max_band: Option<i32>
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
max_files_in_band: Option<i32>
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
additional_options: Option<S3DirectSourceAdditionalOptions>
Specifies additional connection options.
json_path: Option<String>
A JsonPath string defining the JSON data.
multiline: Option<bool>
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is False
, which allows for more aggressive file-splitting during parsing.
output_schemas: Option<Vec<GlueSchema>>
Specifies the data schema for the S3 JSON source.
Implementations§
source§impl S3JsonSource
impl S3JsonSource
sourcepub fn compression_type(&self) -> Option<&CompressionType>
pub fn compression_type(&self) -> Option<&CompressionType>
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip"
and "bzip"
).
sourcepub fn exclusions(&self) -> &[String]
pub fn exclusions(&self) -> &[String]
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "\[\"**.pdf\"\]" excludes all PDF files.
If no value was sent for this field, a default will be set. If you want to determine if no value was sent, use .exclusions.is_none()
.
sourcepub fn group_size(&self) -> Option<&str>
pub fn group_size(&self) -> Option<&str>
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles"
must be set to "inPartition"
for this to take effect.
sourcepub fn group_files(&self) -> Option<&str>
pub fn group_files(&self) -> Option<&str>
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none"
.
sourcepub fn recurse(&self) -> Option<bool>
pub fn recurse(&self) -> Option<bool>
If set to true, recursively reads files in all subdirectories under the specified paths.
sourcepub fn max_band(&self) -> Option<i32>
pub fn max_band(&self) -> Option<i32>
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
sourcepub fn max_files_in_band(&self) -> Option<i32>
pub fn max_files_in_band(&self) -> Option<i32>
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
sourcepub fn additional_options(&self) -> Option<&S3DirectSourceAdditionalOptions>
pub fn additional_options(&self) -> Option<&S3DirectSourceAdditionalOptions>
Specifies additional connection options.
sourcepub fn multiline(&self) -> Option<bool>
pub fn multiline(&self) -> Option<bool>
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is False
, which allows for more aggressive file-splitting during parsing.
sourcepub fn output_schemas(&self) -> &[GlueSchema]
pub fn output_schemas(&self) -> &[GlueSchema]
Specifies the data schema for the S3 JSON source.
If no value was sent for this field, a default will be set. If you want to determine if no value was sent, use .output_schemas.is_none()
.
source§impl S3JsonSource
impl S3JsonSource
sourcepub fn builder() -> S3JsonSourceBuilder
pub fn builder() -> S3JsonSourceBuilder
Creates a new builder-style object to manufacture S3JsonSource
.
Trait Implementations§
source§impl Clone for S3JsonSource
impl Clone for S3JsonSource
source§fn clone(&self) -> S3JsonSource
fn clone(&self) -> S3JsonSource
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moresource§impl Debug for S3JsonSource
impl Debug for S3JsonSource
source§impl PartialEq for S3JsonSource
impl PartialEq for S3JsonSource
source§fn eq(&self, other: &S3JsonSource) -> bool
fn eq(&self, other: &S3JsonSource) -> bool
self
and other
values to be equal, and is used
by ==
.impl StructuralPartialEq for S3JsonSource
Auto Trait Implementations§
impl Freeze for S3JsonSource
impl RefUnwindSafe for S3JsonSource
impl Send for S3JsonSource
impl Sync for S3JsonSource
impl Unpin for S3JsonSource
impl UnwindSafe for S3JsonSource
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
source§impl<T> Instrument for T
impl<T> Instrument for T
source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
source§impl<T> IntoEither for T
impl<T> IntoEither for T
source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moresource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read more