Struct rusoto_sagemaker::S3DataSource
source · [−]pub struct S3DataSource {
pub attribute_names: Option<Vec<String>>,
pub s3_data_distribution_type: Option<String>,
pub s3_data_type: String,
pub s3_uri: String,
}
Expand description
Describes the S3 data source.
Fields
attribute_names: Option<Vec<String>>
A list of one or more attribute names to use that are found in a specified augmented manifest file.
s3_data_distribution_type: Option<String>
If you want Amazon SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify FullyReplicated
.
If you want Amazon SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.
Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (when TrainingInputMode
is set to File
), this copies 1/n of the number of objects.
s3_data_type: String
If you choose S3Prefix
, S3Uri
identifies a key name prefix. Amazon SageMaker uses all objects that match the specified key name prefix for model training.
If you choose ManifestFile
, S3Uri
identifies an object that is a manifest file containing a list of object keys that you want Amazon SageMaker to use for model training.
If you choose AugmentedManifestFile
, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training. AugmentedManifestFile
can only be used if the Channel's input mode is Pipe
.
s3_uri: String
Depending on the value specified for the S3DataType
, identifies either a key name prefix or a manifest. For example:
-
A key name prefix might look like this:
s3://bucketname/exampleprefix
-
A manifest might look like this:
s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of
S3Uri
. Note that the prefix must be a valid non-emptyS3Uri
that precludes users from specifying a manifest whose individualS3Uri
is sourced from different S3 buckets.The following code example shows a valid manifest format:
[ {"prefix": "s3://customerbucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
]
This JSON is equivalent to the following
S3Uri
list:s3://customerbucket/some/prefix/relative/path/to/custdata-1
s3://customerbucket/some/prefix/relative/path/custdata-2
...
s3://customerbucket/some/prefix/relative/path/custdata-N
The complete set of
S3Uri
in this manifest is the input data for the channel for this data source. The object that eachS3Uri
points to must be readable by the IAM role that Amazon SageMaker uses to perform tasks on your behalf.
Trait Implementations
sourceimpl Clone for S3DataSource
impl Clone for S3DataSource
sourcefn clone(&self) -> S3DataSource
fn clone(&self) -> S3DataSource
Returns a copy of the value. Read more
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from source
. Read more
sourceimpl Debug for S3DataSource
impl Debug for S3DataSource
sourceimpl Default for S3DataSource
impl Default for S3DataSource
sourcefn default() -> S3DataSource
fn default() -> S3DataSource
Returns the “default value” for a type. Read more
sourceimpl<'de> Deserialize<'de> for S3DataSource
impl<'de> Deserialize<'de> for S3DataSource
sourcefn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error> where
__D: Deserializer<'de>,
Deserialize this value from the given Serde deserializer. Read more
sourceimpl PartialEq<S3DataSource> for S3DataSource
impl PartialEq<S3DataSource> for S3DataSource
sourcefn eq(&self, other: &S3DataSource) -> bool
fn eq(&self, other: &S3DataSource) -> bool
This method tests for self
and other
values to be equal, and is used
by ==
. Read more
sourcefn ne(&self, other: &S3DataSource) -> bool
fn ne(&self, other: &S3DataSource) -> bool
This method tests for !=
.
sourceimpl Serialize for S3DataSource
impl Serialize for S3DataSource
impl StructuralPartialEq for S3DataSource
Auto Trait Implementations
impl RefUnwindSafe for S3DataSource
impl Send for S3DataSource
impl Sync for S3DataSource
impl Unpin for S3DataSource
impl UnwindSafe for S3DataSource
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
sourceimpl<T> Instrument for T
impl<T> Instrument for T
sourcefn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
sourcefn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
sourceimpl<T> ToOwned for T where
T: Clone,
impl<T> ToOwned for T where
T: Clone,
type Owned = T
type Owned = T
The resulting type after obtaining ownership.
sourcefn clone_into(&self, target: &mut T)
fn clone_into(&self, target: &mut T)
toowned_clone_into
)Uses borrowed data to replace owned data, usually by cloning. Read more
sourceimpl<T> WithSubscriber for T
impl<T> WithSubscriber for T
sourcefn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> where
S: Into<Dispatch>,
fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> where
S: Into<Dispatch>,
Attaches the provided Subscriber
to this type, returning a
WithDispatch
wrapper. Read more
sourcefn with_current_subscriber(self) -> WithDispatch<Self>
fn with_current_subscriber(self) -> WithDispatch<Self>
Attaches the current default Subscriber
to this type, returning a
WithDispatch
wrapper. Read more