Struct DataIntegrationFlowDatasetOptions 

#[non_exhaustive]
pub struct DataIntegrationFlowDatasetOptions {
    pub load_type: Option<DataIntegrationFlowLoadType>,
    pub dedupe_records: Option<bool>,
    pub dedupe_strategy: Option<DataIntegrationFlowDedupeStrategy>,
}

The dataset options used in dataset source and target configurations.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

load_type: Option<DataIntegrationFlowLoadType>

The target dataset's data load type. This only affects how source S3 files are selected in the S3-to-dataset flow.

• REPLACE - Target dataset will get replaced with the new file added under the source s3 prefix.

• INCREMENTAL - Target dataset will get updated with the up-to-date content under S3 prefix incorporating any file additions or removals there.

dedupe_records: Option<bool>

The option to perform deduplication on data records sharing same primary key values. If disabled, transformed data with duplicate primary key values will ingest into dataset, for datasets within asc namespace, such duplicates will cause ingestion fail. If enabled without dedupeStrategy, deduplication is done by retaining a random data record among those sharing the same primary key values. If enabled with dedupeStragtegy, the deduplication is done following the strategy.

Note that target dataset may have partition configured, when dedupe is enabled, it only dedupe against primary keys and retain only one record out of those duplicates regardless of its partition status.

dedupe_strategy: Option<DataIntegrationFlowDedupeStrategy>

The deduplication strategy to dedupe the data records sharing same primary key values of the target dataset. This strategy only applies to target dataset with primary keys and with dedupeRecords option enabled. If transformed data still got duplicates after the dedupeStrategy evaluation, a random data record is chosen to be retained.

Implementations

impl DataIntegrationFlowDatasetOptions

pub fn load_type(&self) -> Option<&DataIntegrationFlowLoadType>

The target dataset's data load type. This only affects how source S3 files are selected in the S3-to-dataset flow.

• REPLACE - Target dataset will get replaced with the new file added under the source s3 prefix.

• INCREMENTAL - Target dataset will get updated with the up-to-date content under S3 prefix incorporating any file additions or removals there.

pub fn dedupe_records(&self) -> Option<bool>

The option to perform deduplication on data records sharing same primary key values. If disabled, transformed data with duplicate primary key values will ingest into dataset, for datasets within asc namespace, such duplicates will cause ingestion fail. If enabled without dedupeStrategy, deduplication is done by retaining a random data record among those sharing the same primary key values. If enabled with dedupeStragtegy, the deduplication is done following the strategy.

Note that target dataset may have partition configured, when dedupe is enabled, it only dedupe against primary keys and retain only one record out of those duplicates regardless of its partition status.

pub fn dedupe_strategy(&self) -> Option<&DataIntegrationFlowDedupeStrategy>

The deduplication strategy to dedupe the data records sharing same primary key values of the target dataset. This strategy only applies to target dataset with primary keys and with dedupeRecords option enabled. If transformed data still got duplicates after the dedupeStrategy evaluation, a random data record is chosen to be retained.

impl DataIntegrationFlowDatasetOptions

pub fn builder() -> DataIntegrationFlowDatasetOptionsBuilder

Creates a new builder-style object to manufacture DataIntegrationFlowDatasetOptions.

Trait Implementations

impl Clone for DataIntegrationFlowDatasetOptions

fn clone(&self) -> DataIntegrationFlowDatasetOptions

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataIntegrationFlowDatasetOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for DataIntegrationFlowDatasetOptions

fn eq(&self, other: &DataIntegrationFlowDatasetOptions) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for DataIntegrationFlowDatasetOptions