#[non_exhaustive]pub struct S3CsvSourceBuilder { /* private fields */ }
Expand description
A builder for S3CsvSource
.
Implementations§
Source§impl S3CsvSourceBuilder
impl S3CsvSourceBuilder
Sourcepub fn name(self, input: impl Into<String>) -> Self
pub fn name(self, input: impl Into<String>) -> Self
The name of the data store.
This field is required.Sourcepub fn paths(self, input: impl Into<String>) -> Self
pub fn paths(self, input: impl Into<String>) -> Self
Appends an item to paths
.
To override the contents of this collection use set_paths
.
A list of the Amazon S3 paths to read from.
Sourcepub fn set_paths(self, input: Option<Vec<String>>) -> Self
pub fn set_paths(self, input: Option<Vec<String>>) -> Self
A list of the Amazon S3 paths to read from.
Sourcepub fn compression_type(self, input: CompressionType) -> Self
pub fn compression_type(self, input: CompressionType) -> Self
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip"
and "bzip"
).
Sourcepub fn set_compression_type(self, input: Option<CompressionType>) -> Self
pub fn set_compression_type(self, input: Option<CompressionType>) -> Self
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip"
and "bzip"
).
Sourcepub fn get_compression_type(&self) -> &Option<CompressionType>
pub fn get_compression_type(&self) -> &Option<CompressionType>
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are "gzip"
and "bzip"
).
Sourcepub fn exclusions(self, input: impl Into<String>) -> Self
pub fn exclusions(self, input: impl Into<String>) -> Self
Appends an item to exclusions
.
To override the contents of this collection use set_exclusions
.
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "\[\"**.pdf\"\]" excludes all PDF files.
Sourcepub fn set_exclusions(self, input: Option<Vec<String>>) -> Self
pub fn set_exclusions(self, input: Option<Vec<String>>) -> Self
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "\[\"**.pdf\"\]" excludes all PDF files.
Sourcepub fn get_exclusions(&self) -> &Option<Vec<String>>
pub fn get_exclusions(&self) -> &Option<Vec<String>>
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "\[\"**.pdf\"\]" excludes all PDF files.
Sourcepub fn group_size(self, input: impl Into<String>) -> Self
pub fn group_size(self, input: impl Into<String>) -> Self
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles"
must be set to "inPartition"
for this to take effect.
Sourcepub fn set_group_size(self, input: Option<String>) -> Self
pub fn set_group_size(self, input: Option<String>) -> Self
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles"
must be set to "inPartition"
for this to take effect.
Sourcepub fn get_group_size(&self) -> &Option<String>
pub fn get_group_size(&self) -> &Option<String>
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files, "groupFiles"
must be set to "inPartition"
for this to take effect.
Sourcepub fn group_files(self, input: impl Into<String>) -> Self
pub fn group_files(self, input: impl Into<String>) -> Self
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none"
.
Sourcepub fn set_group_files(self, input: Option<String>) -> Self
pub fn set_group_files(self, input: Option<String>) -> Self
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none"
.
Sourcepub fn get_group_files(&self) -> &Option<String>
pub fn get_group_files(&self) -> &Option<String>
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to "none"
.
Sourcepub fn recurse(self, input: bool) -> Self
pub fn recurse(self, input: bool) -> Self
If set to true, recursively reads files in all subdirectories under the specified paths.
Sourcepub fn set_recurse(self, input: Option<bool>) -> Self
pub fn set_recurse(self, input: Option<bool>) -> Self
If set to true, recursively reads files in all subdirectories under the specified paths.
Sourcepub fn get_recurse(&self) -> &Option<bool>
pub fn get_recurse(&self) -> &Option<bool>
If set to true, recursively reads files in all subdirectories under the specified paths.
Sourcepub fn max_band(self, input: i32) -> Self
pub fn max_band(self, input: i32) -> Self
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
Sourcepub fn set_max_band(self, input: Option<i32>) -> Self
pub fn set_max_band(self, input: Option<i32>) -> Self
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
Sourcepub fn get_max_band(&self) -> &Option<i32>
pub fn get_max_band(&self) -> &Option<i32>
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
Sourcepub fn max_files_in_band(self, input: i32) -> Self
pub fn max_files_in_band(self, input: i32) -> Self
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
Sourcepub fn set_max_files_in_band(self, input: Option<i32>) -> Self
pub fn set_max_files_in_band(self, input: Option<i32>) -> Self
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
Sourcepub fn get_max_files_in_band(&self) -> &Option<i32>
pub fn get_max_files_in_band(&self) -> &Option<i32>
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
Sourcepub fn additional_options(self, input: S3DirectSourceAdditionalOptions) -> Self
pub fn additional_options(self, input: S3DirectSourceAdditionalOptions) -> Self
Specifies additional connection options.
Sourcepub fn set_additional_options(
self,
input: Option<S3DirectSourceAdditionalOptions>,
) -> Self
pub fn set_additional_options( self, input: Option<S3DirectSourceAdditionalOptions>, ) -> Self
Specifies additional connection options.
Sourcepub fn get_additional_options(&self) -> &Option<S3DirectSourceAdditionalOptions>
pub fn get_additional_options(&self) -> &Option<S3DirectSourceAdditionalOptions>
Specifies additional connection options.
Sourcepub fn separator(self, input: Separator) -> Self
pub fn separator(self, input: Separator) -> Self
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
This field is required.Sourcepub fn set_separator(self, input: Option<Separator>) -> Self
pub fn set_separator(self, input: Option<Separator>) -> Self
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
Sourcepub fn get_separator(&self) -> &Option<Separator>
pub fn get_separator(&self) -> &Option<Separator>
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
Sourcepub fn escaper(self, input: impl Into<String>) -> Self
pub fn escaper(self, input: impl Into<String>) -> Self
Specifies a character to use for escaping. This option is used only when reading CSV files. The default value is none
. If enabled, the character which immediately follows is used as-is, except for a small set of well-known escapes (\n
, \r
, \t
, and \0
).
Sourcepub fn set_escaper(self, input: Option<String>) -> Self
pub fn set_escaper(self, input: Option<String>) -> Self
Specifies a character to use for escaping. This option is used only when reading CSV files. The default value is none
. If enabled, the character which immediately follows is used as-is, except for a small set of well-known escapes (\n
, \r
, \t
, and \0
).
Sourcepub fn get_escaper(&self) -> &Option<String>
pub fn get_escaper(&self) -> &Option<String>
Specifies a character to use for escaping. This option is used only when reading CSV files. The default value is none
. If enabled, the character which immediately follows is used as-is, except for a small set of well-known escapes (\n
, \r
, \t
, and \0
).
Sourcepub fn quote_char(self, input: QuoteChar) -> Self
pub fn quote_char(self, input: QuoteChar) -> Self
Specifies the character to use for quoting. The default is a double quote: '"'
. Set this to -1
to turn off quoting entirely.
Sourcepub fn set_quote_char(self, input: Option<QuoteChar>) -> Self
pub fn set_quote_char(self, input: Option<QuoteChar>) -> Self
Specifies the character to use for quoting. The default is a double quote: '"'
. Set this to -1
to turn off quoting entirely.
Sourcepub fn get_quote_char(&self) -> &Option<QuoteChar>
pub fn get_quote_char(&self) -> &Option<QuoteChar>
Specifies the character to use for quoting. The default is a double quote: '"'
. Set this to -1
to turn off quoting entirely.
Sourcepub fn multiline(self, input: bool) -> Self
pub fn multiline(self, input: bool) -> Self
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is False
, which allows for more aggressive file-splitting during parsing.
Sourcepub fn set_multiline(self, input: Option<bool>) -> Self
pub fn set_multiline(self, input: Option<bool>) -> Self
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is False
, which allows for more aggressive file-splitting during parsing.
Sourcepub fn get_multiline(&self) -> &Option<bool>
pub fn get_multiline(&self) -> &Option<bool>
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is False
, which allows for more aggressive file-splitting during parsing.
Sourcepub fn with_header(self, input: bool) -> Self
pub fn with_header(self, input: bool) -> Self
A Boolean value that specifies whether to treat the first line as a header. The default value is False
.
Sourcepub fn set_with_header(self, input: Option<bool>) -> Self
pub fn set_with_header(self, input: Option<bool>) -> Self
A Boolean value that specifies whether to treat the first line as a header. The default value is False
.
Sourcepub fn get_with_header(&self) -> &Option<bool>
pub fn get_with_header(&self) -> &Option<bool>
A Boolean value that specifies whether to treat the first line as a header. The default value is False
.
Sourcepub fn write_header(self, input: bool) -> Self
pub fn write_header(self, input: bool) -> Self
A Boolean value that specifies whether to write the header to output. The default value is True
.
Sourcepub fn set_write_header(self, input: Option<bool>) -> Self
pub fn set_write_header(self, input: Option<bool>) -> Self
A Boolean value that specifies whether to write the header to output. The default value is True
.
Sourcepub fn get_write_header(&self) -> &Option<bool>
pub fn get_write_header(&self) -> &Option<bool>
A Boolean value that specifies whether to write the header to output. The default value is True
.
Sourcepub fn skip_first(self, input: bool) -> Self
pub fn skip_first(self, input: bool) -> Self
A Boolean value that specifies whether to skip the first data line. The default value is False
.
Sourcepub fn set_skip_first(self, input: Option<bool>) -> Self
pub fn set_skip_first(self, input: Option<bool>) -> Self
A Boolean value that specifies whether to skip the first data line. The default value is False
.
Sourcepub fn get_skip_first(&self) -> &Option<bool>
pub fn get_skip_first(&self) -> &Option<bool>
A Boolean value that specifies whether to skip the first data line. The default value is False
.
Sourcepub fn optimize_performance(self, input: bool) -> Self
pub fn optimize_performance(self, input: bool) -> Self
A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.
Sourcepub fn set_optimize_performance(self, input: Option<bool>) -> Self
pub fn set_optimize_performance(self, input: Option<bool>) -> Self
A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.
Sourcepub fn get_optimize_performance(&self) -> &Option<bool>
pub fn get_optimize_performance(&self) -> &Option<bool>
A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.
Sourcepub fn output_schemas(self, input: GlueSchema) -> Self
pub fn output_schemas(self, input: GlueSchema) -> Self
Appends an item to output_schemas
.
To override the contents of this collection use set_output_schemas
.
Specifies the data schema for the S3 CSV source.
Sourcepub fn set_output_schemas(self, input: Option<Vec<GlueSchema>>) -> Self
pub fn set_output_schemas(self, input: Option<Vec<GlueSchema>>) -> Self
Specifies the data schema for the S3 CSV source.
Sourcepub fn get_output_schemas(&self) -> &Option<Vec<GlueSchema>>
pub fn get_output_schemas(&self) -> &Option<Vec<GlueSchema>>
Specifies the data schema for the S3 CSV source.
Sourcepub fn build(self) -> Result<S3CsvSource, BuildError>
pub fn build(self) -> Result<S3CsvSource, BuildError>
Consumes the builder and constructs a S3CsvSource
.
This method will fail if any of the following fields are not set:
Trait Implementations§
Source§impl Clone for S3CsvSourceBuilder
impl Clone for S3CsvSourceBuilder
Source§fn clone(&self) -> S3CsvSourceBuilder
fn clone(&self) -> S3CsvSourceBuilder
1.0.0 · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moreSource§impl Debug for S3CsvSourceBuilder
impl Debug for S3CsvSourceBuilder
Source§impl Default for S3CsvSourceBuilder
impl Default for S3CsvSourceBuilder
Source§fn default() -> S3CsvSourceBuilder
fn default() -> S3CsvSourceBuilder
Source§impl PartialEq for S3CsvSourceBuilder
impl PartialEq for S3CsvSourceBuilder
impl StructuralPartialEq for S3CsvSourceBuilder
Auto Trait Implementations§
impl Freeze for S3CsvSourceBuilder
impl RefUnwindSafe for S3CsvSourceBuilder
impl Send for S3CsvSourceBuilder
impl Sync for S3CsvSourceBuilder
impl Unpin for S3CsvSourceBuilder
impl UnwindSafe for S3CsvSourceBuilder
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> Paint for Twhere
T: ?Sized,
impl<T> Paint for Twhere
T: ?Sized,
Source§fn fg(&self, value: Color) -> Painted<&T>
fn fg(&self, value: Color) -> Painted<&T>
Returns a styled value derived from self
with the foreground set to
value
.
This method should be used rarely. Instead, prefer to use color-specific
builder methods like red()
and
green()
, which have the same functionality but are
pithier.
§Example
Set foreground color to white using fg()
:
use yansi::{Paint, Color};
painted.fg(Color::White);
Set foreground color to white using white()
.
use yansi::Paint;
painted.white();
Source§fn bright_black(&self) -> Painted<&T>
fn bright_black(&self) -> Painted<&T>
Source§fn bright_red(&self) -> Painted<&T>
fn bright_red(&self) -> Painted<&T>
Source§fn bright_green(&self) -> Painted<&T>
fn bright_green(&self) -> Painted<&T>
Source§fn bright_yellow(&self) -> Painted<&T>
fn bright_yellow(&self) -> Painted<&T>
Source§fn bright_blue(&self) -> Painted<&T>
fn bright_blue(&self) -> Painted<&T>
Source§fn bright_magenta(&self) -> Painted<&T>
fn bright_magenta(&self) -> Painted<&T>
Source§fn bright_cyan(&self) -> Painted<&T>
fn bright_cyan(&self) -> Painted<&T>
Source§fn bright_white(&self) -> Painted<&T>
fn bright_white(&self) -> Painted<&T>
Source§fn bg(&self, value: Color) -> Painted<&T>
fn bg(&self, value: Color) -> Painted<&T>
Returns a styled value derived from self
with the background set to
value
.
This method should be used rarely. Instead, prefer to use color-specific
builder methods like on_red()
and
on_green()
, which have the same functionality but
are pithier.
§Example
Set background color to red using fg()
:
use yansi::{Paint, Color};
painted.bg(Color::Red);
Set background color to red using on_red()
.
use yansi::Paint;
painted.on_red();
Source§fn on_primary(&self) -> Painted<&T>
fn on_primary(&self) -> Painted<&T>
Source§fn on_magenta(&self) -> Painted<&T>
fn on_magenta(&self) -> Painted<&T>
Source§fn on_bright_black(&self) -> Painted<&T>
fn on_bright_black(&self) -> Painted<&T>
Source§fn on_bright_red(&self) -> Painted<&T>
fn on_bright_red(&self) -> Painted<&T>
Source§fn on_bright_green(&self) -> Painted<&T>
fn on_bright_green(&self) -> Painted<&T>
Source§fn on_bright_yellow(&self) -> Painted<&T>
fn on_bright_yellow(&self) -> Painted<&T>
Source§fn on_bright_blue(&self) -> Painted<&T>
fn on_bright_blue(&self) -> Painted<&T>
Source§fn on_bright_magenta(&self) -> Painted<&T>
fn on_bright_magenta(&self) -> Painted<&T>
Source§fn on_bright_cyan(&self) -> Painted<&T>
fn on_bright_cyan(&self) -> Painted<&T>
Source§fn on_bright_white(&self) -> Painted<&T>
fn on_bright_white(&self) -> Painted<&T>
Source§fn attr(&self, value: Attribute) -> Painted<&T>
fn attr(&self, value: Attribute) -> Painted<&T>
Enables the styling Attribute
value
.
This method should be used rarely. Instead, prefer to use
attribute-specific builder methods like bold()
and
underline()
, which have the same functionality
but are pithier.
§Example
Make text bold using attr()
:
use yansi::{Paint, Attribute};
painted.attr(Attribute::Bold);
Make text bold using using bold()
.
use yansi::Paint;
painted.bold();
Source§fn rapid_blink(&self) -> Painted<&T>
fn rapid_blink(&self) -> Painted<&T>
Source§fn quirk(&self, value: Quirk) -> Painted<&T>
fn quirk(&self, value: Quirk) -> Painted<&T>
Enables the yansi
Quirk
value
.
This method should be used rarely. Instead, prefer to use quirk-specific
builder methods like mask()
and
wrap()
, which have the same functionality but are
pithier.
§Example
Enable wrapping using .quirk()
:
use yansi::{Paint, Quirk};
painted.quirk(Quirk::Wrap);
Enable wrapping using wrap()
.
use yansi::Paint;
painted.wrap();
Source§fn clear(&self) -> Painted<&T>
👎Deprecated since 1.0.1: renamed to resetting()
due to conflicts with Vec::clear()
.
The clear()
method will be removed in a future release.
fn clear(&self) -> Painted<&T>
resetting()
due to conflicts with Vec::clear()
.
The clear()
method will be removed in a future release.Source§fn whenever(&self, value: Condition) -> Painted<&T>
fn whenever(&self, value: Condition) -> Painted<&T>
Conditionally enable styling based on whether the Condition
value
applies. Replaces any previous condition.
See the crate level docs for more details.
§Example
Enable styling painted
only when both stdout
and stderr
are TTYs:
use yansi::{Paint, Condition};
painted.red().on_yellow().whenever(Condition::STDOUTERR_ARE_TTY);