dag-ml-data-arrow 0.2.10

Apache Arrow IPC feature buffer reader for dag-ml-data.
Documentation

Apache Arrow IPC feature buffer reader for dag-ml-data.

This crate is a workspace member so the workspace cargo build / cargo test commands always compile it, but the C ABI wiring in dag-ml-data-capi is gated behind the arrow-ipc feature, so the Arrow dependency is opt-in at the cdylib boundary for hosts that ship libdag_ml_data_capi.so. Workspace consumers running the gate still pay the compile-time cost.

The reader maps each Arrow RecordBatch whose schema metadata carries dag_ml_data.feature_set_id to a NumericFeatureMatrixF64Columnar, then resolves the whole stream to a NumericFeatureBufferStore. The mapping is total:

  • Float64Array column → NumericFeatureMatrixF64Columnar.columns[i]
  • Arrow validity bitmap → per-column validity_masks[i]
  • observation_id Utf8Array column → observation_ids
  • feature column names → feature_names

Mandatory schema metadata keys:

  • dag_ml_data.feature_set_id — the feature-set id of the buffer
  • dag_ml_data.representation_id — the representation id of the buffer