Delta Arrow Reader

Delta Arrow Reader is a fast, memory-efficient Delta Lake reader.

• Get results faster. Spend less time downloading and processing data your query doesn't need.
• Use less memory. Process results as they arrive, without keeping the whole result in RAM.
• Skip the cluster setup. Read and filter Delta tables in your own application, without running Spark.

Faster than delta-rs, DuckDB, Polars, and single-machine Spark in all eight of our filtered-read benchmarks, using TPC-H-derived tables with about 60 million rows each. Results compare median query times.

Showing four cases using deletion vectors to mark deleted rows. Full results include all eight cases, with and without deletion vectors.

Get started

Install the reader and Tokio (Rust 1.94 or later). The example uses next_batch(), available on main ahead of the next crates.io release:

cargo add delta-arrow-reader --git https://github.com/mag1cfrog/delta-arrow-reader --branch main
cargo add tokio --features macros,rt-multi-thread

Python bindings are planned; a Python package is not available yet.

Read up to 100 rows from an existing Delta table:

use delta_arrow_reader::DeltaTableBuilder;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let table = DeltaTableBuilder::new("/path/to/delta-table")
        .load_table()
        .await?;
    let scan = table.scan().with_limit(100).build().await?;
    let mut batches = scan.into_stream();

    while let Some(batch) = batches.next_batch().await? {
        println!("Read {} rows", batch.num_rows());
    }
    Ok(())
}

Save this as src/main.rs, replace the table path, and run cargo run.

Read a table or query with SQL. See the Rust API reference for types and methods.

For development and benchmark setup, see the repository documentation.