Macro instrument_with_spans

macro_rules! instrument_with_spans {
    (target: $target:expr, $lvl:expr, options: $options:expr, $($fields:tt)*) => { ... };
    (target: $target:expr, $lvl:expr, options: $options:expr) => { ... };
    ($lvl:expr, options: $options:expr, $($fields:tt)*) => { ... };
    ($lvl:expr, options: $options:expr) => { ... };
}

Constructs a new instrumentation PhysicalOptimizerRule for a DataFusion ExecutionPlan.

This macro is modeled after the span! macro from the tracing crate but is specifically designed to instrument DataFusion execution plans. It creates a new PhysicalOptimizerRule that will wrap each node of the plan in a custom InstrumentedExec node.

By default, it includes all known metrics fields relevant to DataFusion execution.

Instrumentation Options

The instrumentation options are specified via the crate::InstrumentationOptions struct, which includes:

• record_metrics: Enable or disable recording of DataFusion execution metrics.
• preview_limit: Set the number of rows to preview per span (set to 0 to disable).
• preview_fn: Provide an optional callback for formatting previewed batches. If unspecified, datafusion::arrow::util::pretty::pretty_format_batches will be used.
• custom_fields: Provide custom key-value pairs for additional span metadata.

Span Fields

This macro supports the same field–value syntax used by tracing’s span macros, allowing you to add additional contextual information as needed.

Refer to the tracing documentation for more information on the syntax accepted by these macros, as it closely follows span!.

Examples

Creating a new InstrumentRule to wrap the plan with TRACE level spans:

let instrument_rule = instrument_with_spans!(Level::TRACE, options: InstrumentationOptions::default());

Adding additional fields to the instrumentation:

let custom_fields = HashMap::from([
    ("custom.key1".to_string(), "value1".to_string()),
    ("custom.key2".to_string(), "value2".to_string()),
]);
let options = InstrumentationOptions {
   record_metrics: true,
   preview_limit: 10,
   custom_fields,
   ..Default::default()
};
let instrument_rule = instrument_with_spans!(
    Level::INFO,
    options: options,
    datafusion.additional_info = "some info",
    datafusion.user_id = 42,
    custom.key1 = field::Empty,
    custom.key2 = field::Empty,
);
// The instrumentation now includes additional fields, and all spans will be tagged with:
// - "datafusion.additional_info": "some info"
// - "datafusion.user_id": 42
// - "custom.key1": "value1"
// - "custom.key2": "value2"
// as well as all DataFusion metrics fields, and a 10 line preview of the data.

Note for crate Developers:

The list of native datafusion metrics can be re-generated by running the following bash command at the root of the datafusion repository:

(
  find . -type f -name '*.rs' ! -path '*/metrics/mod.rs' -exec grep -A2 'MetricBuilder::new' {} \; | grep -E '(counter|gauge|subset_time)'
  grep -E -o 'Self::.*=>.*"' datafusion/physical-plan/src/metrics/value.rs
) | cut -s -f 2 -d '"' | sort -u | sed 's/\(.*\)/datafusion.metrics.\1 = tracing::field::Empty,/g'