Crate datafusion_openlineage

Expand description

OpenLineage integration for Apache DataFusion.

Instrument a SessionState with OpenLineage::builder to emit OpenLineage run events (START / COMPLETE / FAIL) describing each query’s input/output datasets and column-level lineage. Planning-time work (lineage extraction, context, START) runs in a QueryPlanner; the terminal COMPLETE/FAIL node is installed by a registered ExtensionPlanner that lowers a plan-carried marker — see the rule module and ADR 0005.

The event model, the pluggable Transport seam, and the non-blocking OpenLineageClient live in the engine-agnostic openlineage_client crate; this crate re-exports them and adds the DataFusion-specific glue. Orchestration metadata (parent run, job ids, custom facets) is injected per query via a LineageContextProvider.

§Quickstart

use datafusion::execution::SessionStateBuilder;
use datafusion_openlineage::OpenLineage;

let state = SessionStateBuilder::new_with_default_features().build();
// Reads OPENLINEAGE_URL / _API_KEY / _NAMESPACE / parent-run env vars.
let state = OpenLineage::builder().from_env()?.instrument(state);

Re-exports§

pub use config::DataFusionConfig;
pub use context::LineageContextProvider;
pub use context::StaticContextProvider;
pub use exec::OpenLineageExec;
pub use extract::QueryLineage;
pub use extract::extract;
pub use rule::LineageExtensionPlanner;
pub use rule::LineageHandle;
pub use rule::LineageMarker;
pub use rule::OpenLineageQueryPlanner;
pub use rule::begin_lineage;
pub use session::OpenLineage;
pub use session::OpenLineageBuilder;
pub use session::OpenLineageSqlExt;
pub use session::instrument_session_state;
pub use session::instrument_session_state_simple;

Modules§

client: The OpenLineage client: a non-blocking emit front-end over a Transport.
cloud: Default HTTP transport backed by olai_http::CloudClient.
column: Sound column-level lineage extraction from a DataFusion LogicalPlan.
config: DataFusion engine identity for OpenLineageConfig.
context: The DataFusion-specific context-provider seam.
event: OpenLineage run-event envelope.
exec: Physical-plan wrapper that emits COMPLETE / FAIL at end of execution.
extract: Extract table-level lineage (input/output datasets + schema) from a DataFusion LogicalPlan.
facets: OpenLineage facet types.
naming: OpenLineage dataset naming-spec mapping.
rule: Plan-carried lineage marker and its lowering into the terminal node.
session: The customer-facing entry point: OpenLineage::builder.
transport: Pluggable sink for OpenLineage events.

Structs§

CloudClientTransport: Posts OpenLineage events to an HTTP endpoint via CloudClient.
ConsoleTransport: Logs each event as pretty JSON via tracing. For development/tests.
Dataset: A dataset consumed or produced by a run.
DatasetName: An OpenLineage dataset name: a namespace plus a name unique within it.
Job: A process definition that runs are executions of.
LineageContext: Top-level context an integration contributes to each emitted run.
NoopTransport: Drops events. The safe default when lineage is not configured.
OpenLineageClient: Non-blocking front-end for emitting OpenLineage events.
OpenLineageClientBuilder: Builder for OpenLineageClient.
OpenLineageConfig: Identifies the emitting integration on every emitted event/facet, and tunes the emit transport.
Run: A single execution of a Job, stable across all its events.
RunEvent: A single OpenLineage run event.

Enums§

ClientError: Error returned when an OpenLineageClient cannot be constructed.
RunEventType: Lifecycle stage of a run, per the OpenLineage spec.
TransportError: Error returned when a Transport fails to send an event.

Traits§

Transport: A sink that delivers OpenLineage events to a backend.