Expand description
Serde config types for the partition: block (#479).
A partitioned row is fanned out into N independent invocations, each scoped
to one chunk of a range via ${partition.*} tokens substituted into the
connector configs. This is faucet backfill’s window mechanism generalized:
from time-only to time / integer / offset, and from a separate bounded-replay
command to any run.
§Why the kinds are a tagged enum
Each kind needs a different set of fields, and two of the mistakes are silent data loss rather than errors:
boundsis meaningful forintegerand required there, but meaningless foroffset. As a tagged enum, serde makes it required exactly where it applies instead of a runtime “you must set this when kind is …” check.- A count is not a maximum key.
{"total": 1234567}equals the largest id only when ids are dense and 1-based; chunking an id range from a count silently stops early the moment ids are sparse (deletions, sharded id allocation, non-sequential keys), and every record above it is never fetched. Becausetotallives only onoffsetandtoonly oninteger, that mistake is structurally impossible rather than merely documented.
Structs§
- Bound
Probe - Discover a bound by running a source once and reading one field out of its first record.
Enums§
- Count
Bound - A discoverable row count for an offset range.
- IntBound
- A discoverable upper bound for an integer range.
- Partition
Spec - A partitioned range.
kindselects the variant.