Skip to main content

Module spec

Module spec 

Source
Expand description

Serde config types for the partition: block (#479).

A partitioned row is fanned out into N independent invocations, each scoped to one chunk of a range via ${partition.*} tokens substituted into the connector configs. This is faucet backfill’s window mechanism generalized: from time-only to time / integer / offset, and from a separate bounded-replay command to any run.

§Why the kinds are a tagged enum

Each kind needs a different set of fields, and two of the mistakes are silent data loss rather than errors:

  • bounds is meaningful for integer and required there, but meaningless for offset. As a tagged enum, serde makes it required exactly where it applies instead of a runtime “you must set this when kind is …” check.
  • A count is not a maximum key. {"total": 1234567} equals the largest id only when ids are dense and 1-based; chunking an id range from a count silently stops early the moment ids are sparse (deletions, sharded id allocation, non-sequential keys), and every record above it is never fetched. Because total lives only on offset and to only on integer, that mistake is structurally impossible rather than merely documented.

Structs§

BoundProbe
Discover a bound by running a source once and reading one field out of its first record.

Enums§

CountBound
A discoverable row count for an offset range.
IntBound
A discoverable upper bound for an integer range.
PartitionSpec
A partitioned range. kind selects the variant.