Crate dag_ml 

Public Rust facade for DAG-ML.

Modules

aggregation

bundle

campaign

controller

controller_adapter

Mechanical derivation of ControllerManifests from a thin host-controller descriptor — the dag-ml side of the OperatorController -> ControllerManifest adapter (DEC-CTRL-001 / the “B1” adapter).

controller_registry

Loader for declarative *.controller.yaml registries.

data

dsl

Pipeline DSL: parse, validate, and compile the declarative pipeline spec into a GraphSpec / CompiledPipelineDsl.

error

fold

generation

graph

ids

metrics

observability

ADR-12 observability hooks.

oof

phase

plan

policy

provenance

relation

rng

runtime

Runtime execution: schedulers, controllers, stores, OOF/merge logic.

selection

Structs

AggregateRepresentation

AggregatedPredictionBlock

AggregationControllerResult

AggregationControllerSpec

AggregationControllerTask

AggregationPolicy

ArtifactHandleRecord

ArtifactId

ArtifactInputSpec

ArtifactMaterializationRequest

ArtifactPayloadMaterializationRecord

ArtifactRef

AugmentationPolicy

BranchId

BranchViewPlan

BundleDataRequirement

BundleId

BundlePredictionBlockCacheRecord

BundlePredictionCachePayload

BundlePredictionCachePayloadSet

BundlePredictionCacheRecord

BundlePredictionRequirement

BundleReplayExecution

CampaignFingerprintSpec

CampaignSpec

CandidateScore

CartesianProductRepresentation

ChoiceRef

A reference to a single dimension choice — dimension is a GenerationDimension::name and label a GenerationChoice::label within it. Generation constraints are expressed as sets of these refs; a variant “contains” the ref when its selected choice for dimension carries label.

ColumnarPredictionCacheBlock

ColumnarPredictionCacheManifest

ColumnarPredictionCacheStore

CombinationPlan

CompiledPipelineDsl

ControllerId

ControllerManifest

ControllerRegistry

CrossFoldValidation

The output of cross_fold_validation_reports: the scalar cross-fold OOF average reports (one per producer, fold_id = "avg") plus — purely additively — the per-sample OOF average block + y_true each report was computed from. reports is byte-identical to the historical Vec return; callers that only need the scalars read reports and ignore oof_averages.

DagMlErrorDescriptor

A stable ADR-11 error payload that can be serialized across bindings.

DataBinding

DataHandleRecord

DataMaterializationRequest

DataModelShapePlan

DataOutputProvenance

DataPlan

DataPlanStep

DataProviderViewSpec

DataViewHandleRecord

DataViewPolicy

DataViewRequest

DataViewSelector

EdgeContract

EdgeSpec

EvaluationResult

ExecutionBundle

ExecutionPlan

ExecutionScopePlan

ExplanationBlock

An EXPLAIN-phase output block (ADR-12 explain contract). Explanations are a node output returned in the NodeResult — like predictions, they cross as data, not as an opaque host handle. The payload shape is controller-defined (e.g. per-feature importances); the core does not interpret it. Explanations are only valid in the EXPLAIN phase.

ExternalDataPlanEnvelope

FeatureSelectionPolicy

FileArtifactManifest

Versioned, file-backed artifact manifest. This is a manifest/portability layer only: it records portable ArtifactRef metadata for a bundle’s refit artifacts. It does not deserialize ML objects or materialize artifact payloads; payload stores remain future work.

FileArtifactManifestEntry

One persisted artifact entry. Mirrors the bundle RefitArtifactRecord identity (node, controller, artifact and params fingerprint) while requiring the ArtifactRef to be portable so the manifest stays movable with its payloads.

FileArtifactManifestStore

File-backed artifact manifest store rooted at a directory.

FileArtifactPayloadStore

FilePredictionCacheEntry

FilePredictionCacheManifest

FilePredictionCacheStore

FitInfluenceDiagnostic

FitInfluenceTask

FoldAssignment

FoldId

FoldSet

GenerationChoice

GenerationConstraints

Declarative pruning constraints applied to the enumerated variant set BEFORE materialization.

GenerationDimension

GenerationParamOverride

GenerationSpec

GraphInterface

GraphPlan

GraphSpec

GroupId

GroupKFoldSpec

HandleRef

HostControllerSpec

Host-side description of one OperatorController, the input from which a validated ControllerManifest is mechanically derived.

InMemoryAggregatedPredictionStore

InMemoryArtifactStore

InMemoryDataProvider

InMemoryLineageRecorder

InMemoryPredictionCacheStore

InMemoryPredictionStore

KFoldSpec

LeakageUnitPolicy

LineageId

LineageRecord

ManifestKindTemplate

The mechanical, per-NodeKind portion of a ControllerManifest: the fields a host does not need to author because they follow deterministically from the node kind. HostControllerSpec::derive composes one of these with the host-supplied identity/selectors/overrides.

ModelInputFusionPolicy

ModelInputPortSpec

ModelInputSpec

MonteCarloCartesianRepresentation

NodeId

NodePlan

NodeResult

NodeSpec

NodeTask

ObservationId

ObservationPredictionBlock

OofAverageBlock

The per-sample cross-fold OOF average of one producer, surfaced alongside its scalar report so the host can show each OOF sample’s averaged prediction (nirs4all’s (validation, avg) row y_pred), not only the pooled scalar. The block is keyed by producer_node / partition = Validation / fold_id = "avg" — identical to the scalar RegressionMetricReport this pairs with — and the y_true covers exactly the block’s samples (same id set), so the host pairs them by id. This is REPORT-grade output: it carries no variant tag (the block has none; the variant is stamped on the report downstream) and never feeds a training/feature path, so OOF/leakage invariants are unaffected — it is purely the same averaged values the scalar was computed from, exposed per sample.

OofCampaign

OofMatrix

OpenLineageRunEventOptions

OperatorSelector

OperatorVariantModel

An operator-level variant model lowered from a single operator-level generator (Mechanism B’s PipelineDslStep::Generator): a GenerationDimension whose every choice carries an active_subsequence (the choice’s namespace key) — never param_overrides — paired with the EXACT set of namespaced node ids that choice activates.

ParallelScheduler

PhaseExecutionSchedule

PipelineDslBranch

PipelineDslBranchStep

PipelineDslChoiceRef

PipelineDslConcatBranch

PipelineDslConcatTransformStep

PipelineDslDataPort

PipelineDslGenerationChoice

PipelineDslGenerationConstraints

PipelineDslGenerationDimension

PipelineDslGenerationParamOverride

PipelineDslGeneratorConstraints

Operator-content pruning constraints on a PipelineDslGeneratorStep (ADR-17 1a/1b). Each ref is an operator-content label — a generator branch/option id (_or_) or branch id (_cartesian_), the operator class nirs4all carries in its _mutex_/_requires_/_exclude_. The keywords mirror the nirs4all generation oracle (_generator/constraints.py): mutex = the full group may not all co-occur (issubset), requires = [a, b] means a present requires b present, exclude = [a, b] is a forbidden pair. Compiled to a single-dimension GenerationConstraints over the operator dimension and applied during sequence-build, NOT carried onto OperatorVariantModel.generation_spec.

PipelineDslGeneratorStage

PipelineDslGeneratorStep

PipelineDslMergeModelStep

PipelineDslMergeSelector

PipelineDslMergeStep

PipelineDslOperatorStep

PipelineDslPredictionPort

PipelineDslSequenceStep

PipelineDslShapePlan

PipelineDslSpec

PipelineDslTuningSpec

PipelineDslVariantChoice

PortRef

PortSchema

PortSpec

PredictionBlock

PredictionCacheMaterializationRecord

PredictionCacheMaterializationRequest

PredictionInputSpec

PredictionJoinPolicy

RankedCandidate

ReductionPlan

RefitArtifactRecord

RefitSlotPlan

RegressionMetricReport

RegressionTargetBlock

RegressionTargetRecord

A host-emitted y_true block tagged with the prediction it scores (producer/partition/fold), so the runtime can aggregate ground truth across folds to score cross-fold ensembles natively.

RelationContract

ReplayPhaseRequest

RepresentationComboSelectionRecord

RepresentationCompatibilityReport

RepresentationReplayManifest

RepresentationSampleObservationMapping

ResearchProvenanceExport

ResearchProvenancePackage

ResearchProvenancePackageFile

ResearchProvenancePackageValidation

RunContext

RunId

RuntimeControllerRegistry

SampleId

SampleRelation

SampleRelationSet

SchemaMigrationPolicy

ScoreSet

SeedContext

SelectionDecision

SelectionMetric

SelectionPolicy

SequentialScheduler

ShapeDelta

SplitInvocation

StackFixedRepresentation

StackPaddedMaskedRepresentation

StackingFitContract

StackingOofRefitContract

StackingOofRefitCoverageDiagnostic

StratifiedKFoldSpec

Stratified K-fold: each sample is validated exactly once (OOF-safe like plain K-fold), but folds are balanced by a per-sample class label so every fold mirrors the overall class distribution. strata maps each sample id to its class label (identity-keyed metadata — never feature values).

TargetId

VariantExecutionSpec

VariantId

VariantPlan

VariantSelection

Outcome of native variant selection: the winning variant plus EVERY scored variant’s cross-validation reports, each tagged with its own variant_id.

VariantValidationPredictions

One scored variant’s VALIDATION (OOF) predictions, captured from its transient FIT_CV RunContext and re-tagged with the variant’s id + content fingerprint so a host can fill that variant’s per-sample prediction rows. REPORT-grade output paired with VariantSelection::validation_reports: it never feeds a training/feature path (see the field docs on VariantSelection::variant_validation_predictions).

Enums

AggregationControllerInput

AggregationControllerOutput

AggregationMethod

AggregationWeights

ArtifactBackend

ArtifactPolicy

AugmentationScope

BranchViewMode

CombinationMode

ControllerCapability

ControllerFitScope

DagMlError

DataPlanStepKind

DataRequestPartition

EntityUnitLevel

EvaluationScope

FeatureSelectionScope

FitBoundary

FitInfluenceMechanism

FitInfluencePolicy

FoldPartition

FoldPartitionMode

How validation membership is distributed across a fold set.

GenerationStrategy

Granularity

HandleKind

InferenceFeatures

MetaRowDomain

MetaTrainingFeatures

MetricObjective

MissingnessPolicy

ModelInputFusionMode

NestedCvSpec

Inner (nested) cross-validation policy.

NodeKind

Phase

PipelineDslBranchMode

PipelineDslGeneratorMode

PipelineDslGeneratorValue

PipelineDslMergeOutput

PipelineDslParamGenerator

PipelineDslSelectionSpec

PipelineDslStep

PortCardinality

PortKind

PredictionJoinKey

PredictionLevel

PredictionPartition

PredictionUnitId

ReductionAxis

ReductionMethod

ReductionRole

ReductionTaskCompatibility

RefitStrategy

RegressionMetricKind

RepresentationCardinality

RepresentationCompatibilityOutcome

RepresentationCompatibilitySeverity

RepresentationMissingSourcePolicy

RepresentationPlan

RngPolicy

SelectionProtocol

ShapeDeltaKind

SplitUnit

StackingOofRefitCause

StackingOofRefitDecision

StackingOofRefitPolicy

Constants

AGGREGATION_CONTROLLER_RESULT_SCHEMA_ID

AGGREGATION_CONTROLLER_RESULT_SCHEMA_VERSION

AGGREGATION_CONTROLLER_TASK_SCHEMA_ID

AGGREGATION_CONTROLLER_TASK_SCHEMA_VERSION

BUNDLE_PREDICTION_CACHE_FORMAT

CAMPAIGN_SPEC_SCHEMA_ID

CAMPAIGN_SPEC_SCHEMA_VERSION

CONTROLLER_MANIFEST_SCHEMA_ID

CONTROLLER_MANIFEST_SCHEMA_VERSION

DAGML_OPENLINEAGE_FACET_SCHEMA_URL

DATA_OUTPUT_PROVENANCE_KEY

DATA_OUTPUT_PROVENANCE_SCHEMA_ID

DATA_OUTPUT_PROVENANCE_SCHEMA_VERSION

DATA_PLAN_SCHEMA_ID

DATA_PLAN_SCHEMA_VERSION

DSL_DATA_AWARE_FANOUT_METADATA_KEY

Metadata key under which fan_out_data_aware_branches records the deterministic fingerprint of the discovered partition set, so identical data always expands to a byte-identical spec (and therefore a byte-identical graph/campaign fingerprint downstream).

EXECUTION_BUNDLE_FILE

EXECUTION_BUNDLE_SCHEMA_VERSION

EXECUTION_PLAN_FILE

EXECUTION_PLAN_SCHEMA_ID

EXECUTION_PLAN_SCHEMA_VERSION

EXTERNAL_DATA_PLAN_ENVELOPE_SCHEMA_VERSION

FILE_ARTIFACT_MANIFEST_FILE

FILE_ARTIFACT_MANIFEST_SCHEMA_VERSION

FILE_PREDICTION_CACHE_MANIFEST_FILE

FILE_PREDICTION_CACHE_STORE_SCHEMA_VERSION

GRAPH_SPEC_SCHEMA_ID

GRAPH_SPEC_SCHEMA_VERSION

LINEAGE_RECORDS_FILE

MIN_READABLE_EXECUTION_BUNDLE_SCHEMA_VERSION

MIN_READABLE_PREDICTION_CACHE_PAYLOAD_SCHEMA_VERSION

MIN_WRITABLE_EXECUTION_BUNDLE_SCHEMA_VERSION

MIN_WRITABLE_PREDICTION_CACHE_PAYLOAD_SCHEMA_VERSION

MODEL_INPUT_SPEC_SCHEMA_ID

MODEL_INPUT_SPEC_SCHEMA_VERSION

NODE_RESULT_SCHEMA_ID

NODE_RESULT_SCHEMA_VERSION

NODE_TASK_SCHEMA_ID

NODE_TASK_SCHEMA_VERSION

OBSERVABILITY_FIELD_ALLOWLIST

Frozen ADR-12 telemetry field allowlist. Every field emitted by this module must appear here, and each entry is an identifier or a count — never data. Adding a field requires an ADR-12 update and a review per the privacy rule.

OPENLINEAGE_RUN_EVENT_SCHEMA_URL

PIPELINE_DSL_SCHEMA_ID

PIPELINE_DSL_SCHEMA_VERSION

PREDICTION_CACHE_PAYLOAD_SCHEMA_VERSION

PROV_JSONLD_FILE

REPRESENTATION_TABULAR_NUMERIC

Frozen representation id for a generic numeric feature table — the modality-neutral default stamped on data ports of a derived manifest. Published by the dag-ml-data representation registry (docs/contracts/representation_registry.v1.json, registry id dag-ml-data.representation_registry.v1, type_id = "table").

REPRESENTATION_TARGET_NUMERIC

Frozen representation id for a generic numeric target — the modality-neutral default stamped on target ports of a derived manifest. Published by the same registry (type_id = "target"). This replaces the previous coarse behaviour that (incorrectly) stamped the feature-table id on target ports.

RESEARCH_PROVENANCE_SCHEMA_VERSION

RO_CRATE_METADATA_FILE

SCORE_SET_SCHEMA_VERSION

Current on-disk schema version for ScoreSet.

SELECTION_DECISION_SCHEMA_ID

SELECTION_DECISION_SCHEMA_VERSION

SELECTION_POLICY_SCHEMA_ID

SELECTION_POLICY_SCHEMA_VERSION

SOURCE_INDEX_METADATA_KEY

STACKING_OOF_REFIT_CONTRACT_METADATA_KEY

Traits

RuntimeArtifactStore

RuntimeController

RuntimeDataProvider

RuntimePredictionCacheStore

Functions

aggregate_observation_predictions

aggregate_sample_predictions_by_unit

build_aggregated_prediction_cache_payload

build_aggregated_prediction_cache_record

build_execution_bundle

build_execution_bundle_with_prediction_contracts

build_execution_bundle_with_prediction_requirements

build_execution_plan

build_openlineage_run_event

build_openlineage_run_event_from_package_files

build_prediction_cache_payload

build_prediction_cache_record

build_research_provenance_export

build_research_provenance_package

bundle_prediction_requirement_key

campaign_fingerprint

capabilities_support_fit_influence

compile_operator_variant_models

Opt-in: lower the spec’s operator-level generators (Mechanism B’s PipelineDslStep::Generator) into one OperatorVariantModel each — an active_subsequence-only operator dimension plus the exact per-choice active-node-id set.

compile_pipeline_dsl

compile_pipeline_dsl_with_controller_registry

compile_pipeline_dsl_with_generation

compile_pipeline_dsl_with_generation_and_controller_registry

cross_fold_validation_reports

Score the cross-fold OOF average per producer: concatenate each producer’s per-fold VALIDATION predictions into one block and score it against the combined y_true. Yields one report per producer with fold_id = "avg" — nirs4all’s cv_best_score row — plus, additively, the per-sample OOF average block + y_true each report was computed from (so the host can fill the (validation, avg) row’s per-sample y_pred, not only the scalar). The per-fold join is identity-keyed; producers with a single fold are skipped (nothing to ensemble).

derive_host_controller_registry

Derive every spec and register the manifests into a fresh ControllerRegistry, surfacing the first derivation or duplicate-id error. This is the one call a runtime needs to turn its declared host controllers into a resolvable registry — the replacement for a hardcoded static node registry.

dispatch_custom_aggregation_task

dispatch_custom_observation_aggregation

dispatch_custom_sample_aggregation

emit_oof_refusal

Emit the ADR-12 out-of-fold leakage refusal event with stable taxonomy fields (category/code mirror crate::DagMlError::OofLeakage) so log consumers can alert on refusals without parsing messages.

enumerate_variants

execution_bundle_schema_migration_policy

fan_out_data_aware_branches

Plan-time, data-aware branch fan-out (the keystone of native branch support). Given a parsed pipeline DSL spec and the coordinator data-plan envelope, this turns each auto-separation branch step — a single-template branch over a by_metadata key or by_tag criterion, marked metadata.auto_separate=true — into an explicit branch step with one concrete branch per discovered partition value, retaining ALL of them.

fold_set_fingerprint

generation_spec_fingerprint

join_oof_campaign_features

join_oof_features

load_yaml_manifest_from_path

Load and validate a single manifest from disk.

load_yaml_manifests_from_dir

Walk a directory for *.controller.yaml files and return a deterministically ordered, validated manifest list. Duplicate controller_ids across files are rejected so a directory cannot silently ship two definitions of the same controller.

lower_nirs4all_compat_pipeline_dsl

manifest_kind_template

Return the deterministic manifest defaults for kind.

node_span

Build the per-node span (ADR-12), nested under the current phase span so node telemetry is attributed to its run, plan and controller. Identifiers only.

oof_campaign_fingerprint

operator_variant_canonical_value

Build the canonical serde_json::Value (the JSON array of {"kind", "class", "params"} steps) for a lowered operator sub-sequence — the exact value operator_variant_label hashes. Exposed so a host binding can render the SAME canonical text (and so callers can inspect it in tests).

operator_variant_label

Compute a choice’s variant_label: the cross-language content fingerprint (hex sha256) of a LOWERED operator sub-sequence (Phase 5).

operator_variant_label_from_steps_json

Cross-language entry point: compute the variant_label (hex sha256) of a lowered operator sub-sequence supplied as JSON (steps_json — a JSON array of PipelineDslSteps, the same shape each generator branch carries). The nirs4all host calls THIS through the dag-ml-py binding so it computes the fingerprint over the EXACT SAME canonicalization + serde_json::to_vec (ryu) codepath dag-ml uses to stamp reports — instead of re-deriving it in pure Python, whose json.dumps float formatting diverges from Rust’s for common params (1e-05, 1e-7, 1-ULP shortest decimals). Sharing this one function makes the host label byte-identical to the report label by construction.

parse_pipeline_dsl_json

parse_yaml_manifest

Parse a single YAML manifest from raw text. The same ControllerManifest deserializer is reused so YAML and JSON cannot drift on field shape.

phase_span

Build the per-phase-scope span (ADR-12). run_id/plan_id correlate concurrent or overlapping runs; empty variant_id/fold_id mean the field is not applicable to the current phase. Fields are identifiers only.

plan_oof_partition_mode

The OOF FoldPartitionMode for a plan: its fold set’s mode, or Partition (the clean-OOF default) when the plan carries no fold set. Used to make the cross-fold scoring gate mode-aware so Resampled (ShuffleSplit / repeated CV) campaigns, where a sample is validated in several folds, are not rejected by the Partition exactly-once uniqueness rule.

prediction_cache_payload_schema_migration_policy

prune_plan_to_active

Prune plan (a Mechanism-B operator-generator UNION plan, compiled as a STACKING graph: every choice’s terminal model fans into merge:generator_predictions -> model:meta) down to a single operator-SELECT candidate: the one operator choice in active_nodes plus the prefix it shares with the other choices, with the generator merge + meta-model + every inactive choice physically removed (C Phase 4, #23).

reassemble_merge_targets

Mandatory, central merge target-coverage invariant — the single gate every merge reassembly handler (separation/concat, fusion, off-fold) must pass through before emitting (or declining to emit) a producer-level RegressionTargetBlock for a reassembled merge prediction. It closes audit R-P1-9: a merge that should be scored must never silently produce no score.

reduce_predictions_across_branches

Reduce one prediction block per branch (each from a different producer/model) into a single fused block under merge_node, keyed by sample_id. This is the cross-branch analogue of reduce_predictions_across_folds: where that joins folds of one producer, this joins branches of one merge point. Asymmetric-branch fusion is intrinsic — a modelless branch simply emits no PredictionBlock, so it is absent from branch_blocks and contributes nothing. A sample is averaged over exactly the model-bearing branches that predicted it (its branch coverage), never over a fixed denominator, so partial coverage does not silently down-weight a sample. The join is identity-keyed and the union sample order is first-seen; positional joins are never used.

reduce_predictions_across_folds

Reduce per-fold prediction blocks (same producer + partition) into one block, keyed by sample_id — the native cross-fold ensemble dag-ml previously lacked. A sample that appears in exactly one fold (the disjoint OOF/validation case) passes through unchanged (mean-of-one); a sample predicted by several folds (the shared train/test case) is (weighted) averaged. avg = no weights; w_avg = per-fold weights (e.g. 1/shifted_val_rmse). First-seen sample order is preserved; the join is identity-keyed, never positional.

reduce_proba_mean_across_branches

Probability-mean fusion for classification: average per-class probability rows across branches, keyed by sample_id, under merge_node. Each row of every branch block is treated as a probability vector over the same width classes; rows must be finite, non-negative and sum to 1 (within PROBA_SUM_TOLERANCE). Like reduce_predictions_across_branches this is asymmetric-branch safe — modelless branches contribute no block — and each sample is averaged only over the branches that predicted it. The fused rows are renormalized so each output row is itself a valid probability distribution (it already sums to 1 under equal per-branch weight, but renormalization keeps the contract exact under floating-point and partial coverage).

refit_artifact_input_key

regression_report_to_candidate_score

relation_set_fingerprint

representation_type_id

Return the frozen type_id registered for representation_id, or None when the id is outside the mirrored registry subset ([FROZEN_REPRESENTATION_TYPES]). Hosts use this to type a port while building an explicit data_requirements override; the adapter uses it to synthesize the default one.

resolve_inner_cv

Resolve the effective inner-CV policy for a node: a node-local policy overrides the campaign-global default; None means no nested CV.

resolve_pipeline_dsl_minimal_aliases

score_regression_aggregated_block

score_regression_prediction_block

select_best_operator_variant_by_cv

Pick the best OPERATOR variant of an operator-generator UNION plan by its cross-validation score.

select_best_operator_variant_from_models

Route operator-SELECT from the operator-variant models lowered off a pipeline DSL (compile_operator_variant_models).

select_best_variant_by_cv

Pick the best variant of a multi-variant plan by its cross-validation score, natively.

select_candidate

select_candidate_groups

validate_data_binding_envelope

validate_inner_fold_set_within_outer

Enforce the nested-CV invariant: every sample in inner — both the top-level universe and every fold’s train/validation members — must be an outer-fold training sample (never an outer-validation sample). Holds by construction for dag-ml-built inner folds, and also validates inner folds supplied from elsewhere. Refuses with an OOF-validation error on any leaking sample.

validate_oof_campaign

validate_prediction_blocks_against_folds

validate_prediction_blocks_are_oof

validate_prediction_cache_payload_matches_record

validate_producer_oof_coverage

Mandatory, central OOF coverage invariant — the single gate every path that concatenates a producer’s per-fold validation predictions into one out-of-fold set must pass through. Spec [COORDINATOR_SPEC.md] §“OOF And Leakage Rules” rule 3: every producer must provide exactly one validation prediction per requested sample unless an explicit aggregation policy says otherwise.

validate_research_provenance_package_files

validate_stacking_oof_refit_contract

Type Aliases

Result