Expand description
§faucet-source-pubsub
Google Cloud Pub/Sub source connector for
faucet-stream: streaming
pull from a subscription, per-message record assembly with a configurable
value_format (json / string / bytes) and an attribute map surfaced under
a configurable key, and the standard idle_termination_secs /
max_messages termination knobs (at least one is required).
Delivery is at-least-once. Messages are acked only at durable page
boundaries — a page’s messages are acked once the pipeline has written
that page to the sink and persisted its bookmark, so a crash between the
sink write and the ack redelivers those messages on the next run. Pair with
an upsert sink keyed on message_id when replays must converge. Exactly-once
delivery is out of scope (Pub/Sub provides no compatible primitive).
Structs§
- Pubsub
Bookmark - The persisted (informational) bookmark value.
- Pubsub
Connection - Connection settings shared by the Pub/Sub source and sink. Flattened into
each connector config via
#[serde(flatten)], soproject_id/endpoint/emulator_host/credentialsappear at the config top level. - Pubsub
Source - Google Cloud Pub/Sub source. See the crate README for semantics.
- Pubsub
Source Config - Configuration for
PubsubSource.
Enums§
- Pubsub
Credentials - How to authenticate with Google Cloud Pub/Sub.
- Value
Format - How each message’s
datapayload is decoded into the emitteddatafield.
Constants§
- DEFAULT_
ATTRIBUTES_ KEY - The default JSON key the per-message attribute map is surfaced under.