faucet-source-kinesis 1.0.0

AWS Kinesis Data Streams source connector for faucet-stream (sharded consumer, resumable checkpoints)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86

# faucet-source-kinesis

AWS Kinesis Data Streams **source** connector for
[faucet-stream](https://github.com/PawanSikawat/faucet-stream): consumes
records from every (or selected) shard with bounded per-shard concurrency,
resumable per-shard sequence-number checkpoints, throttle-aware backoff, and
the standard `idle_termination_secs` / `max_messages` termination knobs
(mirroring the Kafka source).

```yaml
source:
  type: kinesis
  config:
    stream_name: events
    region: us-east-1
    start_position: { type: trim_horizon }
    idle_termination_secs: 30      # at least one termination knob is required
    value_format: json
```

## Configuration

| Field | Default | Notes |
|-------|---------|-------|
| `stream_name` | — | Required. |
| `region` | SDK default chain | |
| `endpoint_url` | — | LocalStack / VPC endpoint override. |
| `credentials` | `{ type: default }` | `default` \| `profile` \| `access_key` \| `assume_role` \| `web_identity` — see `faucet-common-kinesis`. |
| `start_position` | `{ type: trim_horizon }` | `trim_horizon` \| `latest` \| `at_timestamp { timestamp_secs }` \| `at_sequence_number { sequence }` \| `after_sequence_number { sequence }`. `at_timestamp` matches at **second** granularity. |
| `shard_ids` | `[]` (all) | Explicit shard allowlist. |
| `include_closed` | `false` | Also drain closed (post-resharding) shards inside the retention window. A fully-drained closed shard counts as *done*, not idle. |
| `poll_interval_ms` | `1000` | Base wait between `GetRecords` per shard; floored at 200 ms (the 5 reads/sec/shard API budget). |
| `records_per_request` | `500` | `GetRecords` limit (1–10000). |
| `shard_concurrency` | `4` | Bounded concurrent shard workers. |
| `idle_termination_secs` / `max_messages` | — | **At least one is required** so a batch run terminates. |
| `value_format` | `json` | `json` (parse; invalid JSON fails with shard+sequence context) \| `string` (strict UTF-8) \| `bytes` (base64). |
| `batch_size` | `1000` | Records per emitted page. `0` = one page per drain cycle. |

## Record shape

```json
{
  "data": { "your": "payload" },
  "partition_key": "user-42",
  "sequence_number": "49593…",
  "shard_id": "shardId-000000000003",
  "approximate_arrival_timestamp_ms": 1716700000123
}
```

## Resume & checkpoints

State key: `kinesis:<stream_name>`. The bookmark is a per-shard map
`{ "shards": { "<shard-id>": "<last-sequence>" } }`, attached cumulatively to
**every** page — any page's bookmark is a valid resume point. On restart a
bookmarked shard resumes at `AFTER_SEQUENCE_NUMBER`; unbookmarked shards use
`start_position`.

**Resharding:** when a parent shard closes mid-run its worker exits cleanly;
child shards are picked up on the next run's discovery and start from
`start_position` (persisted parent sequences prevent re-emitting drained
closed shards when `include_closed: true`).

**Delivery is at-least-once.** A restart between a sink write and the bookmark
persist re-delivers records at the boundary — key downstream on
`sequence_number` (e.g. an upsert sink) when replays must converge.

## Throttling

`ProvisionedThroughputExceededException` (the 2 MB/s/shard read cap) triggers
exponential backoff with per-shard jitter and is never surfaced as an error.
Expired iterators are transparently re-acquired from the last-seen sequence.
Other request failures retry up to 8 times before failing the stream.

## LocalStack

```yaml
config:
  endpoint_url: http://localhost:4566
  region: us-east-1
  credentials: { type: access_key, config: { access_key_id: test, secret_access_key: test } }
```

## License

MIT OR Apache-2.0