Crate deltalake[][src]

Expand description

Native Delta Lake implementation in Rust


Load a Delta Table by path:

async {
  let table = deltalake::open_table("./tests/data/simple_table").await.unwrap();
  let files = table.get_files();

Load a specific version of Delta Table by path then filter files by partitions:

async {
  let table = deltalake::open_table_with_version("./tests/data/simple_table", 0).await.unwrap();
  let files = table.get_files_by_partitions(&[deltalake::PartitionFilter {
      key: "month",
      value: deltalake::PartitionValue::Equal("12"),

Load a specific version of Delta Table by path and datetime:

async {
  let table = deltalake::open_table_with_ds(
  let files = table.get_files();

Optional cargo package features

  • s3 - enable the S3 storage backend to work with Delta Tables in AWS S3.
  • azure - enable the Azure storage backend to work with Delta Tables in Azure Data Lake Storage Gen2 accounts.
  • datafusion-ext - enable the datafusion::datasource::TableProvider trait implementation for Delta Tables, allowing them to be queried using DataFusion.


pub use self::partitions::*;
pub use self::storage::get_backend_for_uri;
pub use self::storage::parse_uri;
pub use self::storage::StorageBackend;
pub use self::storage::StorageError;
pub use self::storage::Uri;
pub use self::storage::UriError;


Actions included in Delta table transaction logs

Implementation for writing delta checkpoints.

Conversion between Delta Table schema and Arrow schema

Delta Table partition handling logic.

Object storage backend abstraction layer for Delta Table transaction logs and data

The writer module contains the DeltaWriter and DeltaWriterError definitions and provides a higher level API for performing Delta transaction writes to a given Delta Table.


Metadata for a checkpoint file

In memory representation of a Delta Table

Delta table metadata

State snapshot currently held by the Delta Table instance.

Object representing a delta transaction. Clients that do not need to mutate action content in case a transaction conflict is encountered may use the commit method and rely on optimistic concurrency to determine the appropriate Delta version number for a commit. A good example of this type of client is an append only client that does not need to maintain transaction state with external systems. Clients that may need to do conflict resolution if the Delta version changes should use the prepare_commit and try_commit_transaction methods and manage the Delta version themselves so that they can resolve data conflicts that may occur between Delta versions.

Options for customizing behavior of a DeltaTransaction

Holds the uri to prepared commit temporary file created with DeltaTransaction.prepare_commit. Once created, the actual commit could be executed with DeltaTransaction.try_commit.

Represents the schema of the delta table.

Describes a specific field of the Delta table schema.

Schema definition for array type fields.

Schema definition for map type fields.

Represents a struct field defined in the Delta table schema.


Error related to Delta log application

Delta Table specific error

Error returned by the DeltaTransaction struct

Error related to checkpoint loading

Enum with variants for each top level schema data type.


Returns rust crate version, can be use used in language bindings to expose Rust core version

Creates and loads a DeltaTable from the given path with current metadata. Infers the storage backend to use from the scheme in the given table path.

Creates a DeltaTable from the given path. Loads metadata from the version appropriate based on the given ISO-8601/RFC-3339 timestamp. Infers the storage backend to use from the scheme in the given table path.

Creates a DeltaTable from the given path and loads it with the metadata from the given version. Infers the storage backend to use from the scheme in the given table path.

Type Definitions

Type alias for i32/Delta int

Type alias for i64/Delta long

Type alias representing the expected type (i64/ms since Unix epoch) of a Delta timestamp.

Type alias representing the expected type (i64) of a Delta table version.

Type alias for a string expected to match a GUID/UUID format