Skip to main content

Module interpolate

Module interpolate 

Source
Expand description

Two-stage interpolation for pipeline configs.

Load-time (interpolate) resolves directives that are knowable before any pipeline has run — environment variables, file contents, secrets. Tokens that don’t match one of these prefixes are left literal so the matrix expander can later treat them as ${row_id.field.path} deferred references.

Record-time (interpolate_record) resolves the remaining ${row_id.dotted.path} tokens against a context map of parent records, producing a string ready to feed into a connector’s Deserialize impl.

Supported load-time directives:

FormResolves to
${env:VAR}the value of environment variable VAR
${file:PATH}the contents of the file at PATH (trimmed)
${secret:VAR}alias for ${env:VAR} (reserved for a future secrets backend)

Anything else (including ${users.id}, ${posts.author.name}) is deferred to record-time. A literal ${ is written $${.

Enums§

Directive
Classification of a ${...} directive body. This is the single rule shared by load-time interpolation, record-time interpolation, and matrix validation (expand.rs) so they can never disagree about what a token means (#78/#39).

Functions§

classify_directive
interpolate
Resolve every load-time directive in input. Unknown prefixes (and tokens with no : at all — i.e. ${row_id.field} references) survive verbatim for record-time resolution.
interpolate_record
Resolve ${id.dotted.path} tokens against ctx. Tokens that look like load-time directives (${env:...}, ${file:...}, ${secret:...}) are left untouched — they should already have been resolved by interpolate.
iter_directives
Iterate every ${...} directive in s, yielding the full token text (including ${ and }) and its Directive classification. $${ is an escape and yields nothing; an unterminated ${ ends iteration. This is the shared tokenizer used by expand.rs validation so it scans and classifies tokens exactly as rewrite does during substitution.
resolve_config_refs
Resolve every ${vars.X}, ${sources.X.PATH}, ${sinks.X.PATH} token found in a parsed PipelineConfig.
resolve_lineage_job_name
Resolve ${name} and ${row_id} in a lineage job-name template. ${now.*} tokens are resolved earlier by the run-clock pass over the config.
resolve_now
Resolve ${now.<token>} references in input against the run clock. Every other ${...} token (env/file/secret/vars/sources/sinks/row-id) is left verbatim. now is a reserved built-in id (see expand.rs). An unknown ${now.<bad>} token is a config error (never silently passed through).