#[non_exhaustive]pub struct S3DataSourceBuilder { /* private fields */ }
Expand description
A builder for S3DataSource
.
Implementations§
Source§impl S3DataSourceBuilder
impl S3DataSourceBuilder
Sourcepub fn s3_data_type(self, input: S3DataType) -> Self
pub fn s3_data_type(self, input: S3DataType) -> Self
If you choose S3Prefix
, S3Uri
identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.
If you choose ManifestFile
, S3Uri
identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.
If you choose AugmentedManifestFile
, S3Uri
identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile
can only be used if the Channel's input mode is Pipe
.
If you choose Converse
, S3Uri
identifies an Amazon S3 location that contains data formatted according to Converse format. This format structures conversational messages with specific roles and content types used for training and fine-tuning foundational models.
Sourcepub fn set_s3_data_type(self, input: Option<S3DataType>) -> Self
pub fn set_s3_data_type(self, input: Option<S3DataType>) -> Self
If you choose S3Prefix
, S3Uri
identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.
If you choose ManifestFile
, S3Uri
identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.
If you choose AugmentedManifestFile
, S3Uri
identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile
can only be used if the Channel's input mode is Pipe
.
If you choose Converse
, S3Uri
identifies an Amazon S3 location that contains data formatted according to Converse format. This format structures conversational messages with specific roles and content types used for training and fine-tuning foundational models.
Sourcepub fn get_s3_data_type(&self) -> &Option<S3DataType>
pub fn get_s3_data_type(&self) -> &Option<S3DataType>
If you choose S3Prefix
, S3Uri
identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.
If you choose ManifestFile
, S3Uri
identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.
If you choose AugmentedManifestFile
, S3Uri
identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile
can only be used if the Channel's input mode is Pipe
.
If you choose Converse
, S3Uri
identifies an Amazon S3 location that contains data formatted according to Converse format. This format structures conversational messages with specific roles and content types used for training and fine-tuning foundational models.
Sourcepub fn s3_uri(self, input: impl Into<String>) -> Self
pub fn s3_uri(self, input: impl Into<String>) -> Self
Depending on the value specified for the S3DataType
, identifies either a key name prefix or a manifest. For example:
-
A key name prefix might look like this:
s3://bucketname/exampleprefix/
-
A manifest might look like this:
s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of
S3Uri
. Note that the prefix must be a valid non-emptyS3Uri
that precludes users from specifying a manifest whose individualS3Uri
is sourced from different S3 buckets.The following code example shows a valid manifest format:
\[ {"prefix": "s3://customer_bucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
\]
This JSON is equivalent to the following
S3Uri
list:s3://customer_bucket/some/prefix/relative/path/to/custdata-1
s3://customer_bucket/some/prefix/relative/path/custdata-2
...
s3://customer_bucket/some/prefix/relative/path/custdata-N
The complete set of
S3Uri
in this manifest is the input data for the channel for this data source. The object that eachS3Uri
points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.
Your input bucket must be located in same Amazon Web Services region as your training job.
This field is required.Sourcepub fn set_s3_uri(self, input: Option<String>) -> Self
pub fn set_s3_uri(self, input: Option<String>) -> Self
Depending on the value specified for the S3DataType
, identifies either a key name prefix or a manifest. For example:
-
A key name prefix might look like this:
s3://bucketname/exampleprefix/
-
A manifest might look like this:
s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of
S3Uri
. Note that the prefix must be a valid non-emptyS3Uri
that precludes users from specifying a manifest whose individualS3Uri
is sourced from different S3 buckets.The following code example shows a valid manifest format:
\[ {"prefix": "s3://customer_bucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
\]
This JSON is equivalent to the following
S3Uri
list:s3://customer_bucket/some/prefix/relative/path/to/custdata-1
s3://customer_bucket/some/prefix/relative/path/custdata-2
...
s3://customer_bucket/some/prefix/relative/path/custdata-N
The complete set of
S3Uri
in this manifest is the input data for the channel for this data source. The object that eachS3Uri
points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.
Your input bucket must be located in same Amazon Web Services region as your training job.
Sourcepub fn get_s3_uri(&self) -> &Option<String>
pub fn get_s3_uri(&self) -> &Option<String>
Depending on the value specified for the S3DataType
, identifies either a key name prefix or a manifest. For example:
-
A key name prefix might look like this:
s3://bucketname/exampleprefix/
-
A manifest might look like this:
s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of
S3Uri
. Note that the prefix must be a valid non-emptyS3Uri
that precludes users from specifying a manifest whose individualS3Uri
is sourced from different S3 buckets.The following code example shows a valid manifest format:
\[ {"prefix": "s3://customer_bucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
\]
This JSON is equivalent to the following
S3Uri
list:s3://customer_bucket/some/prefix/relative/path/to/custdata-1
s3://customer_bucket/some/prefix/relative/path/custdata-2
...
s3://customer_bucket/some/prefix/relative/path/custdata-N
The complete set of
S3Uri
in this manifest is the input data for the channel for this data source. The object that eachS3Uri
points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.
Your input bucket must be located in same Amazon Web Services region as your training job.
Sourcepub fn s3_data_distribution_type(self, input: S3DataDistribution) -> Self
pub fn s3_data_distribution_type(self, input: S3DataDistribution) -> Self
If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated
.
If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode
is set to File
), this copies 1/n of the number of objects.
Sourcepub fn set_s3_data_distribution_type(
self,
input: Option<S3DataDistribution>,
) -> Self
pub fn set_s3_data_distribution_type( self, input: Option<S3DataDistribution>, ) -> Self
If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated
.
If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode
is set to File
), this copies 1/n of the number of objects.
Sourcepub fn get_s3_data_distribution_type(&self) -> &Option<S3DataDistribution>
pub fn get_s3_data_distribution_type(&self) -> &Option<S3DataDistribution>
If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated
.
If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode
is set to File
), this copies 1/n of the number of objects.
Sourcepub fn attribute_names(self, input: impl Into<String>) -> Self
pub fn attribute_names(self, input: impl Into<String>) -> Self
Appends an item to attribute_names
.
To override the contents of this collection use set_attribute_names
.
A list of one or more attribute names to use that are found in a specified augmented manifest file.
Sourcepub fn set_attribute_names(self, input: Option<Vec<String>>) -> Self
pub fn set_attribute_names(self, input: Option<Vec<String>>) -> Self
A list of one or more attribute names to use that are found in a specified augmented manifest file.
Sourcepub fn get_attribute_names(&self) -> &Option<Vec<String>>
pub fn get_attribute_names(&self) -> &Option<Vec<String>>
A list of one or more attribute names to use that are found in a specified augmented manifest file.
Sourcepub fn instance_group_names(self, input: impl Into<String>) -> Self
pub fn instance_group_names(self, input: impl Into<String>) -> Self
Appends an item to instance_group_names
.
To override the contents of this collection use set_instance_group_names
.
A list of names of instance groups that get data from the S3 data source.
Sourcepub fn set_instance_group_names(self, input: Option<Vec<String>>) -> Self
pub fn set_instance_group_names(self, input: Option<Vec<String>>) -> Self
A list of names of instance groups that get data from the S3 data source.
Sourcepub fn get_instance_group_names(&self) -> &Option<Vec<String>>
pub fn get_instance_group_names(&self) -> &Option<Vec<String>>
A list of names of instance groups that get data from the S3 data source.
Sourcepub fn model_access_config(self, input: ModelAccessConfig) -> Self
pub fn model_access_config(self, input: ModelAccessConfig) -> Self
The access configuration file to control access to the ML model. You can explicitly accept the model end-user license agreement (EULA) within the ModelAccessConfig
.
-
If you are a Jumpstart user, see the End-user license agreements section for more details on accepting the EULA.
-
If you are an AutoML user, see the Optional Parameters section of Create an AutoML job to fine-tune text generation models using the API for details on How to set the EULA acceptance when fine-tuning a model using the AutoML API.
Sourcepub fn set_model_access_config(self, input: Option<ModelAccessConfig>) -> Self
pub fn set_model_access_config(self, input: Option<ModelAccessConfig>) -> Self
The access configuration file to control access to the ML model. You can explicitly accept the model end-user license agreement (EULA) within the ModelAccessConfig
.
-
If you are a Jumpstart user, see the End-user license agreements section for more details on accepting the EULA.
-
If you are an AutoML user, see the Optional Parameters section of Create an AutoML job to fine-tune text generation models using the API for details on How to set the EULA acceptance when fine-tuning a model using the AutoML API.
Sourcepub fn get_model_access_config(&self) -> &Option<ModelAccessConfig>
pub fn get_model_access_config(&self) -> &Option<ModelAccessConfig>
The access configuration file to control access to the ML model. You can explicitly accept the model end-user license agreement (EULA) within the ModelAccessConfig
.
-
If you are a Jumpstart user, see the End-user license agreements section for more details on accepting the EULA.
-
If you are an AutoML user, see the Optional Parameters section of Create an AutoML job to fine-tune text generation models using the API for details on How to set the EULA acceptance when fine-tuning a model using the AutoML API.
Sourcepub fn hub_access_config(self, input: HubAccessConfig) -> Self
pub fn hub_access_config(self, input: HubAccessConfig) -> Self
The configuration for a private hub model reference that points to a SageMaker JumpStart public hub model.
Sourcepub fn set_hub_access_config(self, input: Option<HubAccessConfig>) -> Self
pub fn set_hub_access_config(self, input: Option<HubAccessConfig>) -> Self
The configuration for a private hub model reference that points to a SageMaker JumpStart public hub model.
Sourcepub fn get_hub_access_config(&self) -> &Option<HubAccessConfig>
pub fn get_hub_access_config(&self) -> &Option<HubAccessConfig>
The configuration for a private hub model reference that points to a SageMaker JumpStart public hub model.
Sourcepub fn build(self) -> S3DataSource
pub fn build(self) -> S3DataSource
Consumes the builder and constructs a S3DataSource
.
Trait Implementations§
Source§impl Clone for S3DataSourceBuilder
impl Clone for S3DataSourceBuilder
Source§fn clone(&self) -> S3DataSourceBuilder
fn clone(&self) -> S3DataSourceBuilder
1.0.0 · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moreSource§impl Debug for S3DataSourceBuilder
impl Debug for S3DataSourceBuilder
Source§impl Default for S3DataSourceBuilder
impl Default for S3DataSourceBuilder
Source§fn default() -> S3DataSourceBuilder
fn default() -> S3DataSourceBuilder
Source§impl PartialEq for S3DataSourceBuilder
impl PartialEq for S3DataSourceBuilder
impl StructuralPartialEq for S3DataSourceBuilder
Auto Trait Implementations§
impl Freeze for S3DataSourceBuilder
impl RefUnwindSafe for S3DataSourceBuilder
impl Send for S3DataSourceBuilder
impl Sync for S3DataSourceBuilder
impl Unpin for S3DataSourceBuilder
impl UnwindSafe for S3DataSourceBuilder
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moreSource§impl<T> Paint for Twhere
T: ?Sized,
impl<T> Paint for Twhere
T: ?Sized,
Source§fn fg(&self, value: Color) -> Painted<&T>
fn fg(&self, value: Color) -> Painted<&T>
Returns a styled value derived from self
with the foreground set to
value
.
This method should be used rarely. Instead, prefer to use color-specific
builder methods like red()
and
green()
, which have the same functionality but are
pithier.
§Example
Set foreground color to white using fg()
:
use yansi::{Paint, Color};
painted.fg(Color::White);
Set foreground color to white using white()
.
use yansi::Paint;
painted.white();
Source§fn bright_black(&self) -> Painted<&T>
fn bright_black(&self) -> Painted<&T>
Source§fn bright_red(&self) -> Painted<&T>
fn bright_red(&self) -> Painted<&T>
Source§fn bright_green(&self) -> Painted<&T>
fn bright_green(&self) -> Painted<&T>
Source§fn bright_yellow(&self) -> Painted<&T>
fn bright_yellow(&self) -> Painted<&T>
Source§fn bright_blue(&self) -> Painted<&T>
fn bright_blue(&self) -> Painted<&T>
Source§fn bright_magenta(&self) -> Painted<&T>
fn bright_magenta(&self) -> Painted<&T>
Source§fn bright_cyan(&self) -> Painted<&T>
fn bright_cyan(&self) -> Painted<&T>
Source§fn bright_white(&self) -> Painted<&T>
fn bright_white(&self) -> Painted<&T>
Source§fn bg(&self, value: Color) -> Painted<&T>
fn bg(&self, value: Color) -> Painted<&T>
Returns a styled value derived from self
with the background set to
value
.
This method should be used rarely. Instead, prefer to use color-specific
builder methods like on_red()
and
on_green()
, which have the same functionality but
are pithier.
§Example
Set background color to red using fg()
:
use yansi::{Paint, Color};
painted.bg(Color::Red);
Set background color to red using on_red()
.
use yansi::Paint;
painted.on_red();
Source§fn on_primary(&self) -> Painted<&T>
fn on_primary(&self) -> Painted<&T>
Source§fn on_magenta(&self) -> Painted<&T>
fn on_magenta(&self) -> Painted<&T>
Source§fn on_bright_black(&self) -> Painted<&T>
fn on_bright_black(&self) -> Painted<&T>
Source§fn on_bright_red(&self) -> Painted<&T>
fn on_bright_red(&self) -> Painted<&T>
Source§fn on_bright_green(&self) -> Painted<&T>
fn on_bright_green(&self) -> Painted<&T>
Source§fn on_bright_yellow(&self) -> Painted<&T>
fn on_bright_yellow(&self) -> Painted<&T>
Source§fn on_bright_blue(&self) -> Painted<&T>
fn on_bright_blue(&self) -> Painted<&T>
Source§fn on_bright_magenta(&self) -> Painted<&T>
fn on_bright_magenta(&self) -> Painted<&T>
Source§fn on_bright_cyan(&self) -> Painted<&T>
fn on_bright_cyan(&self) -> Painted<&T>
Source§fn on_bright_white(&self) -> Painted<&T>
fn on_bright_white(&self) -> Painted<&T>
Source§fn attr(&self, value: Attribute) -> Painted<&T>
fn attr(&self, value: Attribute) -> Painted<&T>
Enables the styling Attribute
value
.
This method should be used rarely. Instead, prefer to use
attribute-specific builder methods like bold()
and
underline()
, which have the same functionality
but are pithier.
§Example
Make text bold using attr()
:
use yansi::{Paint, Attribute};
painted.attr(Attribute::Bold);
Make text bold using using bold()
.
use yansi::Paint;
painted.bold();
Source§fn rapid_blink(&self) -> Painted<&T>
fn rapid_blink(&self) -> Painted<&T>
Source§fn quirk(&self, value: Quirk) -> Painted<&T>
fn quirk(&self, value: Quirk) -> Painted<&T>
Enables the yansi
Quirk
value
.
This method should be used rarely. Instead, prefer to use quirk-specific
builder methods like mask()
and
wrap()
, which have the same functionality but are
pithier.
§Example
Enable wrapping using .quirk()
:
use yansi::{Paint, Quirk};
painted.quirk(Quirk::Wrap);
Enable wrapping using wrap()
.
use yansi::Paint;
painted.wrap();
Source§fn clear(&self) -> Painted<&T>
👎Deprecated since 1.0.1: renamed to resetting()
due to conflicts with Vec::clear()
.
The clear()
method will be removed in a future release.
fn clear(&self) -> Painted<&T>
resetting()
due to conflicts with Vec::clear()
.
The clear()
method will be removed in a future release.Source§fn whenever(&self, value: Condition) -> Painted<&T>
fn whenever(&self, value: Condition) -> Painted<&T>
Conditionally enable styling based on whether the Condition
value
applies. Replaces any previous condition.
See the crate level docs for more details.
§Example
Enable styling painted
only when both stdout
and stderr
are TTYs:
use yansi::{Paint, Condition};
painted.red().on_yellow().whenever(Condition::STDOUTERR_ARE_TTY);