faucet-source-sqs 1.0.0

AWS SQS source connector for faucet-stream (long-poll receive, at-least-once drain, delete-on-emit)
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

# faucet-source-sqs

AWS SQS **source** connector for
[faucet-stream](https://github.com/PawanSikawat/faucet-stream): long-polls
`ReceiveMessage`, buffers up to `batch_size` messages, and emits them page by
page with bounded memory. It terminates on `idle_timeout_secs` and/or
`max_messages`.

```yaml
source:
  type: sqs
  config:
    queue_url: https://sqs.us-east-1.amazonaws.com/123456789012/events
    region: us-east-1
    idle_timeout_secs: 30      # at least one termination knob is required
    wait_time_seconds: 10
    batch_size: 1000
```

## Configuration

| Field | Default | Notes |
|-------|---------|-------|
| `queue_url` | — | Required. Full SQS queue URL. |
| `region` | SDK default chain | |
| `endpoint_url` | — | LocalStack / VPC endpoint override. |
| `credentials` | `{ type: default }` | `default` \| `profile` \| `access_key` \| `assume_role` \| `web_identity` — see `faucet-common-sqs`. |
| `idle_timeout_secs` / `max_messages` | — | **At least one is required** so a batch run terminates. |
| `wait_time_seconds` | `10` | Long-poll wait per `ReceiveMessage` (0–20). |
| `batch_size` | `1000` | Records per emitted page. `0` = one page for the whole drain. |

Each `ReceiveMessage` call requests up to 10 messages (the SQS API cap),
capped further so it never over-reads past `max_messages`.

## Record shape

Each message body is emitted as its **parsed JSON value** when the body is
valid JSON, otherwise as a JSON **string** of the raw body:

```json
{ "order_id": 42, "status": "shipped" }
```
```json
"a plain, non-JSON body"
```

## Delivery semantics

Each page's receipt handles are deleted (via `DeleteMessageBatch`) right before
the page is yielded. Delivery is **at-least-once**: any message whose delete
does not land — or whose visibility window elapses before a downstream commit —
is redelivered on a later run. There is no resumable bookmark (`bookmark: None`
on every page); the queue itself is the cursor. Key downstream consumers on a
message field (e.g. an upsert sink) when replays must converge.

## LocalStack

```yaml
config:
  endpoint_url: http://localhost:4566
  region: us-east-1
  credentials: { type: access_key, config: { access_key_id: test, secret_access_key: test } }
```

## License

MIT OR Apache-2.0