Skip to main content

Module stage

Module stage 

Source
Expand description

Pipeline-level transform stages. A TransformStage wraps one of four shapes:

apply_stages is the per-record runner: it flat-maps stages left to right, so order matters (a Filter after an Explode filters children). The observability wrapper crate::observability::instrumented_apply_stages calls this per record and aggregates the page-level counters.

Structs§

CompiledExplodetransform-explode
Pre-compiled explode: path parsed once, prefix resolved once.
CompiledFiltertransform-filter
Pre-compiled filter: path parsed once, value cloned once.
CompiledPath
Pre-parsed JSONPath. Bad syntax surfaces at compile_stage time, not per record. The implementation is a hand-rolled parser for the v1 subset (bare key, dot path, bracketed string key) — sufficient because we need to walk parent-container chains for explode mutation, which the jsonpath_rust::query() API doesn’t expose.
ExplodeSpectransform-explode
Spec for TransformStage::Explode. Fans one record out into N records based on the array at path. Compiled into a CompiledExplode at pipeline-build time; per-record work is one path resolve + N clones + N merges.
FilterSpectransform-filter
Predicate spec for TransformStage::Filter. Compiled into a CompiledFilter at pipeline-build time; per-record work is a single path resolve + comparison.

Enums§

CompiledStage
Pre-compiled stage. Per-record work is just lookup + comparison + flat-map.
FilterOptransform-filter
Comparison operator for FilterSpec.
OnMissingtransform-explode
Behaviour when an ExplodeSpec’s path doesn’t yield a non-empty array. The default is Passthrough because silently dropping records is the worst failure mode for ETL pipelines — surfacing the record unchanged lets downstream stages decide.
PathSegment
TransformStage
One stage in a transform pipeline.

Functions§

apply_stages
Per-record stage runner. Returns 0..N output records. Pure; no metrics.
compile_stage
Compile a TransformStage into its CompiledStage form.