Expand description
Pipeline-level transform stages. A TransformStage wraps one of four
shapes:
TransformStage::Mapholds an unchanged 1→1RecordTransform.TransformStage::Filteris a predicate-based 1→0|1 stage (added in Task 4).TransformStage::Explodeexpands an array field into 1→0..N output records (added in Task 5).TransformStage::Customis anFn(Value) -> Vec<Value>closure escape hatch for library callers.
apply_stages is the per-record runner: it flat-maps stages left to
right, so order matters (a Filter after an Explode filters children).
The observability wrapper crate::observability::instrumented_apply_stages
calls this per record and aggregates the page-level counters.
Structs§
- Compiled
Explode transform-explode - Pre-compiled explode: path parsed once, prefix resolved once.
- Compiled
Filter transform-filter - Pre-compiled filter: path parsed once, value cloned once.
- Compiled
Path - Pre-parsed JSONPath. Bad syntax surfaces at
compile_stagetime, not per record. The implementation is a hand-rolled parser for the v1 subset (bare key, dot path, bracketed string key) — sufficient because we need to walk parent-container chains for explode mutation, which thejsonpath_rust::query()API doesn’t expose. - Explode
Spec transform-explode - Spec for
TransformStage::Explode. Fans one record out into N records based on the array atpath. Compiled into aCompiledExplodeat pipeline-build time; per-record work is one path resolve + N clones + N merges. - Filter
Spec transform-filter - Predicate spec for
TransformStage::Filter. Compiled into aCompiledFilterat pipeline-build time; per-record work is a single path resolve + comparison.
Enums§
- Compiled
Stage - Pre-compiled stage. Per-record work is just lookup + comparison + flat-map.
- Filter
Op transform-filter - Comparison operator for
FilterSpec. - OnMissing
transform-explode - Behaviour when an
ExplodeSpec’spathdoesn’t yield a non-empty array. The default isPassthroughbecause silently dropping records is the worst failure mode for ETL pipelines — surfacing the record unchanged lets downstream stages decide. - Path
Segment - Transform
Stage - One stage in a transform pipeline.
Functions§
- apply_
stages - Per-record stage runner. Returns 0..N output records. Pure; no metrics.
- compile_
stage - Compile a
TransformStageinto itsCompiledStageform.