Skip to main content

Module plan

Module plan 

Source
Expand description

Turn a PartitionSpec into concrete chunks and substitute their ${partition.*} tokens into connector configs (#479).

Substitution walks every string leaf of a config Value, exactly like backfill::plan::substitute_unit_tokens — which is why this works for any source without a line of connector code. A REST URL, a SQL WHERE, an object prefix and a Mongo filter are all just strings.

§Injection

Tokens land inside SQL and JSON strings, where substitute_context is documented as unsafe. What makes it safe here is that every rendered value is faucet-generated and type-constrained — an i64, a u64, or an RFC3339 timestamp formatted by chrono — never user data and never a passthrough of anything a remote system said. plan() is the only place token values are produced, and it produces them from typed fields — an i64 start/end, a u64 offset/limit, a chrono-formatted timestamp.

Structs§

PartitionChunk
One planned chunk: a stable id plus the tokens its invocation substitutes.

Functions§

bounds_label
Bounds label for log lines.
has_open_ended
The end predicate for an open-ended final chunk is the caller’s problem — this reports whether one was planned so expand can require the config to handle it.
plan
Plan every chunk for spec.
references_partition
Whether a serialized config references any ${partition.*} token.
substitute
Substitute ${partition.*} tokens in every string leaf of value.