Expand description
Amazon Kinesis Data Firehose is a fully managed service that delivers real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Elasticsearch Service (Amazon ES), Amazon Redshift, and Splunk.
If you’re using the service, you’re probably looking for KinesisFirehoseClient and KinesisFirehose.
Structs§
- Buffering
Hints Describes hints for the buffering to perform before delivering data to the destination. These options are treated as hints, and therefore Kinesis Data Firehose might choose to use different values when it is optimal. The
SizeInMBsandIntervalInSecondsparameters are optional. However, if specify a value for one of them, you must also provide a value for the other.- Cloud
Watch Logging Options Describes the Amazon CloudWatch logging options for your delivery stream.
- Copy
Command Describes a
COPYcommand for Amazon Redshift.- Create
Delivery Stream Input - Create
Delivery Stream Output - Data
Format Conversion Configuration Specifies that you want Kinesis Data Firehose to convert data from the JSON format to the Parquet or ORC format before writing it to Amazon S3. Kinesis Data Firehose uses the serializer and deserializer that you specify, in addition to the column information from the AWS Glue table, to deserialize your input data from JSON and then serialize it to the Parquet or ORC format. For more information, see Kinesis Data Firehose Record Format Conversion.
- Delete
Delivery Stream Input - Delete
Delivery Stream Output - Delivery
Stream Description Contains information about a delivery stream.
- Delivery
Stream Encryption Configuration Contains information about the server-side encryption (SSE) status for the delivery stream, the type customer master key (CMK) in use, if any, and the ARN of the CMK. You can get
DeliveryStreamEncryptionConfigurationby invoking the DescribeDeliveryStream operation.- Delivery
Stream Encryption Configuration Input Specifies the type and Amazon Resource Name (ARN) of the CMK to use for Server-Side Encryption (SSE).
- Describe
Delivery Stream Input - Describe
Delivery Stream Output - Deserializer
The deserializer you want Kinesis Data Firehose to use for converting the input data from JSON. Kinesis Data Firehose then serializes the data to its final format using the Serializer. Kinesis Data Firehose supports two types of deserializers: the Apache Hive JSON SerDe and the OpenX JSON SerDe.
- Destination
Description Describes the destination for a delivery stream.
- Elasticsearch
Buffering Hints Describes the buffering to perform before delivering data to the Amazon ES destination.
- Elasticsearch
Destination Configuration Describes the configuration of a destination in Amazon ES.
- Elasticsearch
Destination Description The destination description in Amazon ES.
- Elasticsearch
Destination Update Describes an update for a destination in Amazon ES.
- Elasticsearch
Retry Options Configures retry behavior in case Kinesis Data Firehose is unable to deliver documents to Amazon ES.
- Encryption
Configuration Describes the encryption for a destination in Amazon S3.
- Extended
S3Destination Configuration Describes the configuration of a destination in Amazon S3.
- Extended
S3Destination Description Describes a destination in Amazon S3.
- Extended
S3Destination Update Describes an update for a destination in Amazon S3.
- Failure
Description Provides details in case one of the following operations fails due to an error related to KMS: CreateDeliveryStream, DeleteDeliveryStream, StartDeliveryStreamEncryption, StopDeliveryStreamEncryption.
- Hive
Json SerDe The native Hive / HCatalog JsonSerDe. Used by Kinesis Data Firehose for deserializing data, which means converting it from the JSON format in preparation for serializing it to the Parquet or ORC format. This is one of two deserializers you can choose, depending on which one offers the functionality you need. The other option is the OpenX SerDe.
- Http
Endpoint Buffering Hints Describes the buffering options that can be applied before data is delivered to the HTTP endpoint destination. Kinesis Data Firehose treats these options as hints, and it might choose to use more optimal values. The
SizeInMBsandIntervalInSecondsparameters are optional. However, if specify a value for one of them, you must also provide a value for the other.- Http
Endpoint Common Attribute Describes the metadata that's delivered to the specified HTTP endpoint destination.
- Http
Endpoint Configuration Describes the configuration of the HTTP endpoint to which Kinesis Firehose delivers data.
- Http
Endpoint Description Describes the HTTP endpoint selected as the destination.
- Http
Endpoint Destination Configuration Describes the configuration of the HTTP endpoint destination.
- Http
Endpoint Destination Description Describes the HTTP endpoint destination.
- Http
Endpoint Destination Update Updates the specified HTTP endpoint destination.
- Http
Endpoint Request Configuration The configuration of the HTTP endpoint request.
- Http
Endpoint Retry Options Describes the retry behavior in case Kinesis Data Firehose is unable to deliver data to the specified HTTP endpoint destination, or if it doesn't receive a valid acknowledgment of receipt from the specified HTTP endpoint destination.
- Input
Format Configuration Specifies the deserializer you want to use to convert the format of the input data. This parameter is required if
Enabledis set to true.- KMSEncryption
Config Describes an encryption key for a destination in Amazon S3.
- Kinesis
Firehose Client - A client for the Firehose API.
- Kinesis
Stream Source Configuration The stream and role Amazon Resource Names (ARNs) for a Kinesis data stream used as the source for a delivery stream.
- Kinesis
Stream Source Description Details about a Kinesis data stream used as the source for a Kinesis Data Firehose delivery stream.
- List
Delivery Streams Input - List
Delivery Streams Output - List
Tags ForDelivery Stream Input - List
Tags ForDelivery Stream Output - OpenX
Json SerDe The OpenX SerDe. Used by Kinesis Data Firehose for deserializing data, which means converting it from the JSON format in preparation for serializing it to the Parquet or ORC format. This is one of two deserializers you can choose, depending on which one offers the functionality you need. The other option is the native Hive / HCatalog JsonSerDe.
- OrcSer
De A serializer to use for converting data to the ORC format before storing it in Amazon S3. For more information, see Apache ORC.
- Output
Format Configuration Specifies the serializer that you want Kinesis Data Firehose to use to convert the format of your data before it writes it to Amazon S3. This parameter is required if
Enabledis set to true.- Parquet
SerDe A serializer to use for converting data to the Parquet format before storing it in Amazon S3. For more information, see Apache Parquet.
- Processing
Configuration Describes a data processing configuration.
- Processor
Describes a data processor.
- Processor
Parameter Describes the processor parameter.
- PutRecord
Batch Input - PutRecord
Batch Output - PutRecord
Batch Response Entry Contains the result for an individual record from a PutRecordBatch request. If the record is successfully added to your delivery stream, it receives a record ID. If the record fails to be added to your delivery stream, the result includes an error code and an error message.
- PutRecord
Input - PutRecord
Output - Record
The unit of data in a delivery stream.
- Redshift
Destination Configuration Describes the configuration of a destination in Amazon Redshift.
- Redshift
Destination Description Describes a destination in Amazon Redshift.
- Redshift
Destination Update Describes an update for a destination in Amazon Redshift.
- Redshift
Retry Options Configures retry behavior in case Kinesis Data Firehose is unable to deliver documents to Amazon Redshift.
- S3Destination
Configuration Describes the configuration of a destination in Amazon S3.
- S3Destination
Description Describes a destination in Amazon S3.
- S3Destination
Update Describes an update for a destination in Amazon S3.
- Schema
Configuration Specifies the schema to which you want Kinesis Data Firehose to configure your data before it writes it to Amazon S3. This parameter is required if
Enabledis set to true.- Serializer
The serializer that you want Kinesis Data Firehose to use to convert data to the target format before writing it to Amazon S3. Kinesis Data Firehose supports two types of serializers: the ORC SerDe and the Parquet SerDe.
- Source
Description Details about a Kinesis data stream used as the source for a Kinesis Data Firehose delivery stream.
- Splunk
Destination Configuration Describes the configuration of a destination in Splunk.
- Splunk
Destination Description Describes a destination in Splunk.
- Splunk
Destination Update Describes an update for a destination in Splunk.
- Splunk
Retry Options Configures retry behavior in case Kinesis Data Firehose is unable to deliver documents to Splunk, or if it doesn't receive an acknowledgment from Splunk.
- Start
Delivery Stream Encryption Input - Start
Delivery Stream Encryption Output - Stop
Delivery Stream Encryption Input - Stop
Delivery Stream Encryption Output - Tag
Metadata that you can assign to a delivery stream, consisting of a key-value pair.
- TagDelivery
Stream Input - TagDelivery
Stream Output - Untag
Delivery Stream Input - Untag
Delivery Stream Output - Update
Destination Input - Update
Destination Output - VpcConfiguration
The details of the VPC of the Amazon ES destination.
- VpcConfiguration
Description The details of the VPC of the Amazon ES destination.
Enums§
- Create
Delivery Stream Error - Errors returned by CreateDeliveryStream
- Delete
Delivery Stream Error - Errors returned by DeleteDeliveryStream
- Describe
Delivery Stream Error - Errors returned by DescribeDeliveryStream
- List
Delivery Streams Error - Errors returned by ListDeliveryStreams
- List
Tags ForDelivery Stream Error - Errors returned by ListTagsForDeliveryStream
- PutRecord
Batch Error - Errors returned by PutRecordBatch
- PutRecord
Error - Errors returned by PutRecord
- Start
Delivery Stream Encryption Error - Errors returned by StartDeliveryStreamEncryption
- Stop
Delivery Stream Encryption Error - Errors returned by StopDeliveryStreamEncryption
- TagDelivery
Stream Error - Errors returned by TagDeliveryStream
- Untag
Delivery Stream Error - Errors returned by UntagDeliveryStream
- Update
Destination Error - Errors returned by UpdateDestination
Traits§
- Kinesis
Firehose - Trait representing the capabilities of the Firehose API. Firehose clients implement this trait.