Crate ruststream_kinesis 

Amazon Kinesis Data Streams broker for the RustStream messaging framework, backed by the official aws-sdk-kinesis client.

A subscription reads every shard of one stream, and a publish is one record on the stream the outgoing message names. The crate adds what the vendor’s consumer library gives other platforms: shard discovery across splits and merges, a lease per shard, and per-shard checkpoints in a LeaseStore. An ack is a checkpoint, one instance reads a shard at a time, and delivery is at-least-once.

A service

A handler is an async fn over the decoded record, mounted on a broker inside the application object, and the attribute writes the main:

use ruststream_kinesis::prelude::*;
use serde::Deserialize;

#[derive(Deserialize)]
struct Order {
    id: u64,
}

#[subscriber(KinesisStream::new("orders"))]
async fn handle(order: &Order) -> HandlerOutcome {
    tracing::info!(order.id, "accepted");
    HandlerOutcome::ack()
}

#[ruststream::app]
fn app() -> impl App {
    RustStream::new(AppInfo::new("orders", "0.1.0")).with_broker(KinesisBroker::new(), |b| {
        b.include(handle);
    })
}

cargo run -- run starts it. KinesisBroker::new only records the configuration; the region and the credentials resolve from the environment when the runtime connects the broker, and the runtime shuts it down when the service stops.

Where things are

The guides take a Kinesis service from installation to production, in this order: getting started, consuming a stream, shards, partition keys and ordering, checkpoints and delivery guarantees, retries and dead-letter streams, replay and seeking, several instances, testing and running in production. The architecture pages show how the crate sits between the SDK client and the runtime, and the benchmarks what it costs.

Each topic below links to its component page on the site, which shows it in a service; the item’s own documentation keeps the detail.

Broker

• KinesisBroker: KinesisBroker, connected into ConnectedKinesisBroker.
• Credentials: the AWS default chain at connect, or a config built elsewhere through KinesisBroker::from_config.
• Endpoint: KinesisBroker::endpoint, KinesisBroker::region and KinesisBroker::test_credentials, for a local stack.
• Startup: a subscription opens once every shard it took has a reader.
• Shutdown: a handle used after shutdown reports KinesisError::NotConnected.
• Prelude: prelude, the one glob a Kinesis service imports.
• Errors: KinesisError.

Subscriptions

• Subscriptions: a bare stream name reads every shard of the stream.
• KinesisStream: KinesisStream, one stream with every setting of its subscription.
• poll_interval: KinesisStream::poll_interval, the pause between reads of an idle shard.
• create_if_missing: KinesisStream::create_if_missing, a stream created when it is missing.
• KinesisSubscriberExt: KinesisSubscriberExt, the same settings on the mount chain.
• Shards: a lease and a reader per shard, and a split’s children after their parents.
• Batches: a slice handler over KinesisSubscriber batches.

Settlement

• Checkpoint: KinesisMessage, what an ack, a drop and a requeue do on a shard.
• retry_after: a deferred copy of the record back to its stream.
• out_retry: the publisher the retry copies leave through.
• Dead letters: max_attempts and dead_letter, the stream a spent record goes to.

Leases

• LeaseStore: LeaseStore, LeaseState and SHARD_END, where shard owners and checkpoints live.
• MemoryLeaseStore: MemoryLeaseStore, the default store for one instance.
• DynamoLeaseStore: DynamoLeaseStore, a DynamoDB table several instances share (feature dynamodb-lease).
• LeaseKey: LeaseKey, one shard of one stream.
• owner_id: KinesisBroker::owner_id, the name this instance holds its leases under.
• legacy_stream: DynamoLeaseStore::legacy_stream, a lease table an earlier version wrote.

Context

• KinesisContext: KinesisContext and its keys Position and SeekHandle.
• KinesisBatchContext: KinesisBatchContext, a batch handler’s context.
• Headers: SEQUENCE_HEADER, SHARD_HEADER and PARTITION_KEY_HEADER on every delivery.
• Envelope: how user headers travel inside a record.

Positions

• KinesisPosition: KinesisPosition, a place in the stream.
• start_at: opening a subscription at a position on every startup.
• Seek: KinesisSeeker through the SeekHandle key, moving a running subscription.

Publishing

• KinesisPublish: KinesisPublish, the publish policy, and its live KinesisPublisher.
• Keys: the four places that name a record’s partition key.
• partition_key: KinesisPublishSteps::partition_key and KinesisPublishOptions key one record.
• KinesisPublishSettings: KinesisPublishSettings::partition_key keys every record of a position.

Tooling

• AsyncAPI: the Kinesis bindings of the generated document (feature asyncapi): the server, the channels and the messages.
• Testing: the in-process Kinesis TestApp connects the broker to (feature testing), and Live runs against LocalStack or AWS.

Cargo features

All three are additive and off by default. asyncapi contributes the Kinesis bindings to the generated document, dynamodb-lease adds DynamoLeaseStore, and testing adds the in-process mode TestApp::start connects the broker through. The codecs are the core’s: a service enables json, msgpack or cbor on ruststream.

Modules

prelude

The imports a service on Kinesis writes every time, in one glob.

Structs

ConnectedKinesisBroker

The typed witness that connect succeeded: holds the live SDK client directly.

DynamoLeaseStore

A DynamoDB-backed lease store.

KinesisBatchContext

This broker’s batch context: the subscription’s reposition handle, shared by every record of the batch.

KinesisBroker

An Amazon Kinesis Data Streams broker for the RustStream messaging framework.

KinesisContext

This broker’s per-delivery context: where the record sits in the log, and the handle that repositions the subscription it arrived on.

KinesisMessage

A record delivered by a KinesisSubscriber.

KinesisPublish

The publish policy for KinesisPublisher: pure declaration, constructible anywhere, paired with the connected broker by the runtime after connect.

KinesisPublishOptions

The per-message publish settings of this broker: what one call varies, against what the mount site fixed on the policy.

KinesisPublisher

Publishes records to Kinesis streams; the destination is the stream’s name.

KinesisSeeker

Repositions a KinesisSubscriber while its stream runs; minted by Seekable::seeker.

KinesisStream

A subscription descriptor for one Kinesis stream.

KinesisSubscriber

A subscription to one Kinesis stream; yields KinesisMessages from every owned shard.

LeaseKey

The name one shard’s lease and checkpoint live under: the stream and the shard id.

LeaseState

The persisted state of one shard’s lease.

MemoryLeaseStore

In-process lease coordination, correct for a single service instance.

Position

The key reading this record’s KinesisPosition out of KinesisContext.

SeekHandle

The key reading the subscription’s KinesisSeeker out of either context.

Enums

KinesisError

Errors returned by the Amazon Kinesis Data Streams broker.

KinesisPosition

A position in the stream’s retained log: the whole start vocabulary of this broker, accepted by Seeker::seek and by the start_at(..) clause of #[subscriber(..)].

Constants

PARTITION_KEY_HEADER

Header carrying the partition key, mapped onto the record’s own partition key.

SEQUENCE_HEADER

Header exposing the record’s sequence number; every delivery carries it.

SHARD_END

The checkpoint value marking a shard fully consumed; its children may start.

SHARD_HEADER

Header exposing the shard a record arrived on; every delivery carries it.

Traits

KinesisPublishSettings

The Kinesis publish settings, chained on a mount site after the position that named the policy: .out_reply(policy), .out_retry(policy) or .out(marker, policy).

KinesisPublishSteps

The Kinesis publish steps, on the framework’s publish builder.

KinesisSubscriberExt

The Kinesis subscription settings, chained on a mount site after the framework’s own.

LeaseStore

Durable coordination for shard leases and checkpoints.

Type Aliases

LeaseError

The boxed error lease stores report; the crate wraps it with the stream and the shard for diagnostics.