openlineage_client/lib.rs
1//! Transport-agnostic [OpenLineage](https://openlineage.io) event model and emit client.
2//!
3//! This crate owns the emission side of OpenLineage: the [`RunEvent`] model and
4//! its typed facets, a pluggable [`Transport`] sink, and a non-blocking
5//! [`OpenLineageClient`] that hands events to a background drain task. It has
6//! **no DataFusion (or any engine) dependency** — engine integrations such as
7//! [`datafusion-openlineage`](https://docs.rs/datafusion-openlineage) build on
8//! top of it, and so can any other emitter.
9//!
10//! # The transport seam
11//!
12//! How events are published is deliberately unspecified. The [`Transport`] trait
13//! is the only seam: an implementation might POST to a spec-compliant OpenLineage
14//! REST endpoint, publish to a Kafka topic, or do anything else. The crate ships
15//! [`NoopTransport`], [`ConsoleTransport`], and — behind the `http` feature —
16//! [`CloudClientTransport`], which posts to an (optionally authenticated) HTTP
17//! endpoint via `olai-http`. To target something else, implement [`Transport`].
18//!
19//! # Non-blocking emission
20//!
21//! [`OpenLineageClient::emit`] never blocks and never applies back-pressure: it
22//! hands the event to a bounded channel drained by a background task. If the
23//! queue is full the event is dropped with a warning — lineage must never slow or
24//! break the host workload. Call [`OpenLineageClient::shutdown`] before exit to
25//! drain queued events and flush the transport.
26//!
27//! # Example
28//!
29//! ```no_run
30//! use std::sync::Arc;
31//! use openlineage_client::{ConsoleTransport, OpenLineageClient};
32//!
33//! # async fn run(event: openlineage_client::RunEvent) {
34//! let client = OpenLineageClient::new(Arc::new(ConsoleTransport));
35//! client.emit(event); // non-blocking
36//! client.shutdown().await; // drain + flush before exit
37//! # }
38//! ```
39//!
40//! # Environment
41//!
42//! [`OpenLineageClient::from_env`] and [`OpenLineageConfig::from_env`] read the
43//! standard OpenLineage environment variables, so an integration can wire itself
44//! up from the environment the rest of the ecosystem already uses:
45//!
46//! | Variable | Read by | Meaning |
47//! |---|---|---|
48//! | `OPENLINEAGE_URL` | [`OpenLineageClient::from_env`] | Base URL of the endpoint (unset → no-op client) |
49//! | `OPENLINEAGE_ENDPOINT` | [`OpenLineageClient::from_env`] | Path appended to the URL (default `/api/v1/lineage`) |
50//! | `OPENLINEAGE_API_KEY` | [`OpenLineageClient::from_env`] | Bearer token, if set |
51//! | `OPENLINEAGE_NAMESPACE` | [`OpenLineageConfig::from_env`] | Default job namespace |
52//! | `OPENLINEAGE_TIMEOUT_MS` | [`OpenLineageConfig::from_env`] | Per-request transport timeout |
53//! | `OPENLINEAGE_PARENT_ID` / `OPENLINEAGE_PARENT_*` | [`LineageContext::from_env`] | Parent run facet |
54#![deny(missing_docs)]
55
56pub mod client;
57pub mod config;
58pub mod context;
59pub mod event;
60pub mod facets;
61pub mod naming;
62pub mod transport;
63
64#[cfg(feature = "http")]
65pub mod cloud;
66
67pub use client::{ClientError, OpenLineageClient, OpenLineageClientBuilder};
68pub use config::{DEFAULT_NAMESPACE, DEFAULT_PRODUCER, DEFAULT_REQUEST_TIMEOUT, OpenLineageConfig};
69pub use context::LineageContext;
70pub use event::{Dataset, Job, Run, RunEvent, RunEventType};
71pub use naming::DatasetName;
72pub use transport::{ConsoleTransport, NoopTransport, Transport, TransportError};
73
74#[cfg(feature = "http")]
75pub use cloud::CloudClientTransport;