Crate delta_arrow_reader

Expand description

§delta-arrow-reader

delta-arrow-reader is a read-only Delta Lake reader for Apache Arrow record batches. It provides a direct pull-driven stream API and an optional DataFusion table provider. The caller owns the Tokio runtime, and scans do not collect the whole result in memory.

§Installation

The 0.1.0 package declaration is:

[dependencies]
delta-arrow-reader = "0.1.0"
futures-util = "0.3"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Enable the DataFusion adapter and add the matching DataFusion dependency when you need SQL integration:

[dependencies]
datafusion = { version = "54.1.0", default-features = false, features = ["sql"] }
delta-arrow-reader = { version = "0.1.0", features = ["datafusion"] }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

§Read a table directly

Use load_async from asynchronous code. The returned stream exposes live scan metrics and yields batches as the caller requests them.

use delta_arrow_reader::{DeltaComparison, DeltaPredicate, DeltaScalar, DeltaTableBuilder};
use futures_util::TryStreamExt;

let table = DeltaTableBuilder::new("/tmp/example-delta-table")
    .load_async()
    .await?;
let scan = table
    .scan()
    .with_projection(vec!["id".into(), "name".into()])
    .with_predicate(DeltaPredicate::Compare {
        column: "id".into(),
        op: DeltaComparison::GtEq,
        value: DeltaScalar::Int64(10),
    })
    .with_limit(100)
    .build()
    .await?;
let mut batches = scan.execute().await?;
let metrics = batches.metrics();

while let Some(batch) = batches.try_next().await? {
    println!("rows={}", batch.num_rows());
}

println!("files={}", metrics.snapshot().files_completed);

DeltaTableBuilder::load is the blocking table-load alternative. Scan building and execution remain asynchronous.

§Register a DataFusion table

The datafusion feature exposes DeltaTableProvider, register_delta_table, and DataFusion execution metrics. Registration loads no data files; reads begin when DataFusion executes a query.

use datafusion::prelude::SessionContext;
use delta_arrow_reader::{
    DeltaDataFusionScanOptions, DeltaTableBuilder, register_delta_table,
};

let context = SessionContext::new();
let table = DeltaTableBuilder::new("/tmp/example-delta-table")
    .load_async()
    .await?;
register_delta_table(
    &context,
    "orders",
    table,
    DeltaDataFusionScanOptions::default(),
)?;

let batches = context.sql("SELECT * FROM orders").await?.collect().await?;
println!("batches={}", batches.len());

§Features

Feature	Default	Purpose
`native-async`	Yes	Native asynchronous Parquet data-file reader and I/O metrics.
`official-kernel`	No	Official Delta Kernel data-file reader backend.
`datafusion`	No	DataFusion provider, registration, filtering, execution, and metrics.

At least one reader backend must be enabled to execute a scan. Select the backend for a scan with DeltaReaderExecutionOptions::with_reader_backend. When default features are enabled, NativeAsync is selected by default.

§Runtime, errors, and metrics

The caller supplies the Tokio runtime and drives returned streams.
Execution limits, buffering, Parquet metadata prefetch, and optional full-file reads are configured through DeltaReaderExecutionOptions.
DeltaReaderError::phase and DeltaReaderError::as_str return stable, redacted categories. Dependency failures remain available through the standard error source chain.
DeltaReadMetrics is a cloneable live handle. snapshot returns an immutable point-in-time view. NativeAsync Parquet I/O counters are None when the OfficialKernel backend is selected.

§Scope

The crate supports the extracted read path: snapshot selection, protocol and schema loading, projections, predicates, deletion vectors, partition planning, bounded scheduling, NativeAsync and OfficialKernel data-file reads, and the optional DataFusion adapter.

It does not write Delta tables, manage transactions, create a Tokio runtime, or provide Delta Funnel orchestration, reporting, or Python APIs.

See architecture, provenance, and the security policy for repository details.

§Development checks

The repository CI runs every feature combination. The focused local checks are:

cargo test --locked --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --locked --all-features --no-deps
cargo package --locked

Structs§

DeltaBatchStream: Pull-driven stream of finalized logical Arrow batches from one Delta scan.
DeltaDataFusionMetrics: Shared live metrics for one DataFusion physical scan plan.
DeltaDataFusionMetricsSnapshot: Immutable point-in-time DataFusion scan metrics.
DeltaDataFusionScanOptions: DataFusion-specific scan settings for one provider.
DeltaProtocolInfo: Protocol metadata captured from one immutable Delta snapshot.
DeltaReadMetrics: Shared live metrics for one Delta scan.
DeltaReadMetricsSnapshot: Immutable point-in-time metrics for one Delta scan.
DeltaReaderExecutionOptions: Bounded execution settings for one Delta scan.
DeltaScan: One immutable, single-use direct Delta scan plan.
DeltaScanBuilder: Configures one single-use direct Delta scan.
DeltaTable: One immutable loaded Delta table snapshot.
DeltaTableBuilder: Configures and loads one immutable Delta table snapshot.
DeltaTableProvider: Immutable DataFusion provider for one loaded Delta table snapshot.
RegisteredDeltaTable: Result of registering one loaded Delta table in a DataFusion context.

Enums§

DeltaComparison: Comparison operation in a Delta predicate.
DeltaPredicate: Query-engine-neutral Delta predicate.
DeltaReaderBackend: Backend used to read Delta data files.
DeltaReaderError: Redacted failure returned by reader APIs.
DeltaReaderPhase: Reader operation phase associated with an error.
DeltaScalar: Non-null scalar value in a Delta predicate.
DeltaSnapshotSelection: Delta snapshot selected for a table load.

Constants§

VERSION: The crate version.

Functions§

collect_delta_datafusion_metrics: Collects distinct Delta DataFusion scan metrics in depth-first plan order.
register_delta_table: Registers one loaded Delta table in a DataFusion session.
version: Returns the crate version.

Type Aliases§

DeltaStorageOptions: Storage options forwarded to Delta object-store construction.