Crate delta_arrow_reader

Expand description

§Delta Arrow Reader

Delta Lake in. Arrow batches out. No Spark required.

Delta Arrow Reader is a fast, memory-efficient Delta Lake reader.

Get results faster. Spend less time downloading and processing data your query doesn’t need.
Use less memory. Process results as they arrive, without keeping the whole result in RAM.
Skip the cluster setup. Read and filter Delta tables in your own application, without running Spark.

Faster than delta-rs, DuckDB, Polars, and single-machine Spark in all eight of our filtered-read benchmarks, using TPC-H-derived tables with about 60 million rows each. Results compare median query times.

_{Showing four cases using deletion vectors to mark deleted rows. Full results include all eight cases, with and without deletion vectors.}

§Get started

Install the reader and Tokio (Rust 1.94 or later). The example uses next_batch(), available on main ahead of the next crates.io release:

cargo add delta-arrow-reader --git https://github.com/mag1cfrog/delta-arrow-reader --branch main
cargo add tokio --features macros,rt-multi-thread

Python bindings are planned; a Python package is not available yet.

Read up to 100 rows from an existing Delta table:

use delta_arrow_reader::DeltaTableBuilder;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let table = DeltaTableBuilder::new("/path/to/delta-table")
        .load_table()
        .await?;
    let scan = table.scan().with_limit(100).build().await?;
    let mut batches = scan.into_stream();

    while let Some(batch) = batches.next_batch().await? {
        println!("Read {} rows", batch.num_rows());
    }
    Ok(())
}

Save this as src/main.rs, replace the table path, and run cargo run.

Read a table or query with SQL. See the Rust API reference for types and methods.

For development and benchmark setup, see the repository documentation.

Modules§

datafusion: Optional DataFusion table-provider and registration surface.
guides: Guides for getting started and understanding how the reader works.

Structs§

DeltaBatchStream: Pull-driven stream of finalized logical Arrow batches from one Delta scan.
DeltaProtocol: Protocol metadata captured from one immutable Delta snapshot.
DeltaScan: One immutable, single-use streaming Delta scan plan.
DeltaScanBuilder: Configures one single-use streaming Delta scan.
DeltaScanExecutionOptions: Bounded execution settings for one Delta scan.
DeltaScanMetrics: Shared live metrics for one Delta scan.
DeltaScanMetricsSnapshot: Immutable point-in-time copy of one Delta scan’s metrics.
DeltaTable: One immutable loaded Delta table snapshot.
DeltaTableBuilder: Configures and loads one immutable Delta table snapshot.
DeltaTableSnapshot: Loaded Delta snapshot metadata awaiting logical Arrow schema conversion.

Enums§

DeltaComparison: Comparison operation in a Delta predicate.
DeltaPredicate: Query-engine-neutral Delta predicate.
DeltaReaderError: Redacted failure returned by reader APIs.
DeltaReaderPhase: Reader operation phase associated with an error.
DeltaScalar: Non-null scalar value in a Delta predicate.
DeltaSnapshotSelection: Delta snapshot selected for a table load.
ParquetReaderBackend: Backend used to read Parquet data files.
WarmupMode: Selects how much work table loading performs before returning.

Constants§

VERSION: The crate version.

Traits§

TryStreamExt: Adapters specific to Result-returning streams

Type Aliases§

DeltaStorageOptions: Storage options forwarded to Delta object-store construction.