Struct aws_sdk_glue::types::KinesisStreamingSourceOptions

#[non_exhaustive]
pub struct KinesisStreamingSourceOptions {

    pub endpoint_url: Option<String>,
    pub stream_name: Option<String>,
    pub classification: Option<String>,
    pub delimiter: Option<String>,
    pub starting_position: Option<StartingPosition>,
    pub max_fetch_time_in_ms: Option<i64>,
    pub max_fetch_records_per_shard: Option<i64>,
    pub max_record_per_read: Option<i64>,
    pub add_idle_time_between_reads: Option<bool>,
    pub idle_time_between_reads_in_ms: Option<i64>,
    pub describe_shard_interval: Option<i64>,
    pub num_retries: Option<i32>,
    pub retry_interval_ms: Option<i64>,
    pub max_retry_interval_ms: Option<i64>,
    pub avoid_empty_batches: Option<bool>,
    pub stream_arn: Option<String>,
    pub role_arn: Option<String>,
    pub role_session_name: Option<String>,
    pub add_record_timestamp: Option<String>,
    pub emit_consumer_lag_metrics: Option<String>,
    pub starting_timestamp: Option<DateTime>,
}

Additional options for the Amazon Kinesis streaming data source.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

endpoint_url: Option<String>

The URL of the Kinesis endpoint.

stream_name: Option<String>

The name of the Kinesis data stream.

classification: Option<String>

An optional classification.

delimiter: Option<String>

Specifies the delimiter character.

starting_position: Option<StartingPosition>

The starting position in the Kinesis data stream to read data from. The possible values are "latest", "trim_horizon", "earliest", or a timestamp string in UTC format in the pattern yyyy-mm-ddTHH:MM:SSZ (where Z represents a UTC timezone offset with a +/-. For example: "2023-04-04T08:00:00-04:00"). The default value is "latest".

Note: Using a value that is a timestamp string in UTC format for "startingPosition" is supported only for Glue version 4.0 or later.

max_fetch_time_in_ms: Option<i64>

The maximum time spent for the job executor to read records for the current batch from the Kinesis data stream, specified in milliseconds (ms). Multiple GetRecords API calls may be made within this time. The default value is 1000.

max_fetch_records_per_shard: Option<i64>

The maximum number of records to fetch per shard in the Kinesis data stream per microbatch. Note: The client can exceed this limit if the streaming job has already read extra records from Kinesis (in the same get-records call). If MaxFetchRecordsPerShard needs to be strict then it needs to be a multiple of MaxRecordPerRead. The default value is 100000.

max_record_per_read: Option<i64>

The maximum number of records to fetch from the Kinesis data stream in each getRecords operation. The default value is 10000.

add_idle_time_between_reads: Option<bool>

Adds a time delay between two consecutive getRecords operations. The default value is "False". This option is only configurable for Glue version 2.0 and above.

idle_time_between_reads_in_ms: Option<i64>

The minimum time delay between two consecutive getRecords operations, specified in ms. The default value is 1000. This option is only configurable for Glue version 2.0 and above.

describe_shard_interval: Option<i64>

The minimum time interval between two ListShards API calls for your script to consider resharding. The default value is 1s.

num_retries: Option<i32>

The maximum number of retries for Kinesis Data Streams API requests. The default value is 3.

retry_interval_ms: Option<i64>

The cool-off time period (specified in ms) before retrying the Kinesis Data Streams API call. The default value is 1000.

max_retry_interval_ms: Option<i64>

The maximum cool-off time period (specified in ms) between two retries of a Kinesis Data Streams API call. The default value is 10000.

avoid_empty_batches: Option<bool>

Avoids creating an empty microbatch job by checking for unread data in the Kinesis data stream before the batch is started. The default value is "False".

stream_arn: Option<String>

The Amazon Resource Name (ARN) of the Kinesis data stream.

role_arn: Option<String>

The Amazon Resource Name (ARN) of the role to assume using AWS Security Token Service (AWS STS). This role must have permissions for describe or read record operations for the Kinesis data stream. You must use this parameter when accessing a data stream in a different account. Used in conjunction with "awsSTSSessionName".

role_session_name: Option<String>

An identifier for the session assuming the role using AWS STS. You must use this parameter when accessing a data stream in a different account. Used in conjunction with "awsSTSRoleARN".

add_record_timestamp: Option<String>

When this option is set to 'true', the data output will contain an additional column named "__src_timestamp" that indicates the time when the corresponding record received by the stream. The default value is 'false'. This option is supported in Glue version 4.0 or later.

emit_consumer_lag_metrics: Option<String>

When this option is set to 'true', for each batch, it will emit the metrics for the duration between the oldest record received by the stream and the time it arrives in Glue to CloudWatch. The metric's name is "glue.driver.streaming.maxConsumerLagInMs". The default value is 'false'. This option is supported in Glue version 4.0 or later.

starting_timestamp: Option<DateTime>

The timestamp of the record in the Kinesis data stream to start reading data from. The possible values are a timestamp string in UTC format of the pattern yyyy-mm-ddTHH:MM:SSZ (where Z represents a UTC timezone offset with a +/-. For example: "2023-04-04T08:00:00+08:00").

Implementations

impl KinesisStreamingSourceOptions

pub fn endpoint_url(&self) -> Option<&str>

The URL of the Kinesis endpoint.

pub fn stream_name(&self) -> Option<&str>

The name of the Kinesis data stream.

pub fn classification(&self) -> Option<&str>

An optional classification.

pub fn delimiter(&self) -> Option<&str>

Specifies the delimiter character.

pub fn starting_position(&self) -> Option<&StartingPosition>

The starting position in the Kinesis data stream to read data from. The possible values are "latest", "trim_horizon", "earliest", or a timestamp string in UTC format in the pattern yyyy-mm-ddTHH:MM:SSZ (where Z represents a UTC timezone offset with a +/-. For example: "2023-04-04T08:00:00-04:00"). The default value is "latest".

Note: Using a value that is a timestamp string in UTC format for "startingPosition" is supported only for Glue version 4.0 or later.

pub fn max_fetch_time_in_ms(&self) -> Option<i64>

The maximum time spent for the job executor to read records for the current batch from the Kinesis data stream, specified in milliseconds (ms). Multiple GetRecords API calls may be made within this time. The default value is 1000.

pub fn max_fetch_records_per_shard(&self) -> Option<i64>

The maximum number of records to fetch per shard in the Kinesis data stream per microbatch. Note: The client can exceed this limit if the streaming job has already read extra records from Kinesis (in the same get-records call). If MaxFetchRecordsPerShard needs to be strict then it needs to be a multiple of MaxRecordPerRead. The default value is 100000.

pub fn max_record_per_read(&self) -> Option<i64>

The maximum number of records to fetch from the Kinesis data stream in each getRecords operation. The default value is 10000.

pub fn add_idle_time_between_reads(&self) -> Option<bool>

Adds a time delay between two consecutive getRecords operations. The default value is "False". This option is only configurable for Glue version 2.0 and above.

pub fn idle_time_between_reads_in_ms(&self) -> Option<i64>

The minimum time delay between two consecutive getRecords operations, specified in ms. The default value is 1000. This option is only configurable for Glue version 2.0 and above.

pub fn describe_shard_interval(&self) -> Option<i64>

The minimum time interval between two ListShards API calls for your script to consider resharding. The default value is 1s.

pub fn num_retries(&self) -> Option<i32>

The maximum number of retries for Kinesis Data Streams API requests. The default value is 3.

pub fn retry_interval_ms(&self) -> Option<i64>

The cool-off time period (specified in ms) before retrying the Kinesis Data Streams API call. The default value is 1000.

pub fn max_retry_interval_ms(&self) -> Option<i64>

The maximum cool-off time period (specified in ms) between two retries of a Kinesis Data Streams API call. The default value is 10000.

pub fn avoid_empty_batches(&self) -> Option<bool>

Avoids creating an empty microbatch job by checking for unread data in the Kinesis data stream before the batch is started. The default value is "False".

pub fn stream_arn(&self) -> Option<&str>

The Amazon Resource Name (ARN) of the Kinesis data stream.

pub fn role_arn(&self) -> Option<&str>

The Amazon Resource Name (ARN) of the role to assume using AWS Security Token Service (AWS STS). This role must have permissions for describe or read record operations for the Kinesis data stream. You must use this parameter when accessing a data stream in a different account. Used in conjunction with "awsSTSSessionName".

pub fn role_session_name(&self) -> Option<&str>

An identifier for the session assuming the role using AWS STS. You must use this parameter when accessing a data stream in a different account. Used in conjunction with "awsSTSRoleARN".

pub fn add_record_timestamp(&self) -> Option<&str>

When this option is set to 'true', the data output will contain an additional column named "__src_timestamp" that indicates the time when the corresponding record received by the stream. The default value is 'false'. This option is supported in Glue version 4.0 or later.

pub fn emit_consumer_lag_metrics(&self) -> Option<&str>

When this option is set to 'true', for each batch, it will emit the metrics for the duration between the oldest record received by the stream and the time it arrives in Glue to CloudWatch. The metric's name is "glue.driver.streaming.maxConsumerLagInMs". The default value is 'false'. This option is supported in Glue version 4.0 or later.

pub fn starting_timestamp(&self) -> Option<&DateTime>

The timestamp of the record in the Kinesis data stream to start reading data from. The possible values are a timestamp string in UTC format of the pattern yyyy-mm-ddTHH:MM:SSZ (where Z represents a UTC timezone offset with a +/-. For example: "2023-04-04T08:00:00+08:00").

impl KinesisStreamingSourceOptions

pub fn builder() -> KinesisStreamingSourceOptionsBuilder

Creates a new builder-style object to manufacture KinesisStreamingSourceOptions.

Trait Implementations

impl Clone for KinesisStreamingSourceOptions

fn clone(&self) -> KinesisStreamingSourceOptions

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KinesisStreamingSourceOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for KinesisStreamingSourceOptions

fn eq(&self, other: &KinesisStreamingSourceOptions) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for KinesisStreamingSourceOptions