faucet-source-kinesis 1.0.0

AWS Kinesis Data Streams source connector for faucet-stream (sharded consumer, resumable checkpoints)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#![cfg_attr(docsrs, feature(doc_cfg))]

//! # faucet-source-kinesis
//!
//! AWS Kinesis Data Streams source connector for
//! [faucet-stream](https://github.com/PawanSikawat/faucet-stream): consumes
//! records from every (or selected) shard with bounded per-shard concurrency,
//! resumable per-shard sequence-number checkpoints, throttle-aware backoff,
//! and the standard `idle_termination_secs` / `max_messages` termination
//! knobs (mirroring the Kafka source).
//!
//! Delivery is **at-least-once**: a restart between a sink write and the
//! bookmark persist re-delivers records at the boundary. Pair with an upsert
//! sink keyed on `sequence_number` when replays must converge.

mod config;
mod shard;
mod state;
mod stream;

pub use config::{KinesisSourceConfig, StartPosition, ValueFormat};
pub use state::ShardBookmarks;
pub use stream::KinesisSource;

// Shared connection types, re-exported so users need only this crate.
pub use faucet_common_kinesis::{KinesisCredentials, build_client};