Crate delta_arrow_reader

Expand description

§Delta Arrow Reader

Delta Lake in. Arrow batches out. No Spark required.

Delta Arrow Reader is a fast, memory-efficient Delta Lake reader.

Get results faster. Spend less time downloading and processing data your query doesn’t need.
Use less memory. Process results as they arrive, without keeping the whole result in RAM.
Skip the cluster setup. Read and filter Delta tables in your own application, without running Spark.

Faster than delta-rs, DuckDB, Polars, and single-machine Spark in all eight of our filtered-read benchmarks, using TPC-H-derived tables with about 60 million rows each. Results compare median times for the first complete query after table initialization, excluding startup.

_{Showing four cases using deletion vectors to mark deleted rows. Full results include all eight cases, with and without deletion vectors.}

§Get started

Install the reader and Tokio (Rust 1.94 or later):

cargo add delta-arrow-reader
cargo add tokio --features macros,rt-multi-thread

For Python, install a CI wheel or build from source to read tables with PyArrow. The same package includes the native delta-arrow-reader CLI. Install the wheel with pip for both the API and command, or use uv tool/uvx for a separate CLI environment. PyPI publishing is planned.

For Go and other languages, install a version-pinned dar binary or container. The CLI returns snapshot metadata as JSON and rows as an Arrow IPC stream. The Go example reads that stream through a subprocess without cgo.

Read up to 100 rows from an existing Delta table:

use delta_arrow_reader::DeltaTableBuilder;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let table = DeltaTableBuilder::new("/path/to/delta-table")
        .load_table()
        .await?;
    let scan = table.scan().with_limit(100).build().await?;
    let mut batches = scan.into_stream();

    while let Some(batch) = batches.next_batch().await? {
        println!("Read {} rows", batch.num_rows());
    }
    Ok(())
}

Save this as src/main.rs, replace the table path, and run cargo run.

Read a table or query with SQL. See the Rust API reference for types and methods.

For development and benchmark setup, see the repository documentation.

Modules§

datafusion: Optional DataFusion table-provider and registration surface.
guides: Guides for getting started and understanding how the reader works.

Structs§

DeltaBatchStream: Pull-driven stream of finalized logical Arrow batches from one Delta scan.
DeltaProtocol: Protocol metadata captured from one immutable Delta snapshot.
DeltaScan: One immutable, single-use streaming Delta scan plan.
DeltaScanBuilder: Configures one single-use streaming Delta scan.
DeltaScanExecutionOptions: Bounded execution settings for one Delta scan.
DeltaScanMetrics: Shared live metrics for one Delta scan.
DeltaScanMetricsSnapshot: Immutable point-in-time copy of one Delta scan’s metrics.
DeltaTable: One immutable loaded Delta table snapshot.
DeltaTableBuilder: Configures and loads one immutable Delta table snapshot.
DeltaTableSnapshot: Loaded Delta snapshot metadata awaiting logical Arrow schema conversion.

Enums§

DeltaComparison: Comparison operation in a Delta predicate.
DeltaPredicate: Query-engine-neutral Delta predicate.
DeltaReaderError: Redacted failure returned by reader APIs.
DeltaReaderPhase: Reader operation phase associated with an error.
DeltaScalar: Non-null scalar value in a Delta predicate.
DeltaSnapshotSelection: Delta snapshot selected for a table load.
ParquetReaderBackend: Backend used to read Parquet data files.
WarmupMode: Selects how much work table loading performs before returning.

Constants§

VERSION: The crate version.

Traits§

TryStreamExt: Adapters specific to Result-returning streams

Type Aliases§

DeltaStorageOptions: Storage options forwarded to Delta object-store construction.