Crate deltalake

Native Delta Lake implementation in Rust

Usage

Load a Delta Table by path:

async {
  let table = deltalake::open_table("./tests/data/simple_table").await.unwrap();
  let files = table.get_files();
};

Load a specific version of Delta Table by path then filter files by partitions:

async {
  let table = deltalake::open_table_with_version("./tests/data/simple_table", 0).await.unwrap();
  let files = table.get_files_by_partitions(&[deltalake::PartitionFilter {
      key: "month",
      value: deltalake::PartitionValue::Equal("12"),
  }]);
};

Load a specific version of Delta Table by path and datetime:

async {
  let table = deltalake::open_table_with_ds(
      "./tests/data/simple_table",
      "2020-05-02T23:47:31-07:00",
  ).await.unwrap();
  let files = table.get_files();
};

Optional cargo package features

• s3 - enable the S3 storage backend to work with Delta Tables in AWS S3.
• azure - enable the Azure storage backend to work with Delta Tables in Azure Data Lake Storage Gen2 accounts.
• datafusion-ext - enable the datafusion::datasource::TableProvider trait implementation for Delta Tables, allowing them to be queried using DataFusion.

Re-exports

pub use self::partitions::*;

pub use self::storage::get_backend_for_uri;

pub use self::storage::parse_uri;

pub use self::storage::StorageBackend;

pub use self::storage::StorageError;

pub use self::storage::Uri;

pub use self::storage::UriError;

Modules

action

Actions included in Delta table transaction logs

checkpoints

Implementation for writing delta checkpoints.

delta_arrow

Conversion between Delta Table schema and Arrow schema

partitions

Delta Table partition handling logic.

storage

Object storage backend abstraction layer for Delta Table transaction logs and data

writer

The writer module contains the DeltaWriter and DeltaWriterError definitions and provides a higher level API for performing Delta transaction writes to a given Delta Table.

Structs

CheckPoint

Metadata for a checkpoint file

DeltaTable

In memory representation of a Delta Table

DeltaTableMetaData

Delta table metadata

DeltaTableState

State snapshot currently held by the Delta Table instance.

DeltaTransaction

Object representing a delta transaction. Clients that do not need to mutate action content in case a transaction conflict is encountered may use the commit method and rely on optimistic concurrency to determine the appropriate Delta version number for a commit. A good example of this type of client is an append only client that does not need to maintain transaction state with external systems. Clients that may need to do conflict resolution if the Delta version changes should use the prepare_commit and try_commit_transaction methods and manage the Delta version themselves so that they can resolve data conflicts that may occur between Delta versions.

DeltaTransactionOptions

Options for customizing behavior of a DeltaTransaction

PreparedCommit

Holds the uri to prepared commit temporary file created with DeltaTransaction.prepare_commit. Once created, the actual commit could be executed with DeltaTransaction.try_commit.

Schema

Represents the schema of the delta table.

SchemaField

Describes a specific field of the Delta table schema.

SchemaTypeArray

Schema definition for array type fields.

SchemaTypeMap

Schema definition for map type fields.

SchemaTypeStruct

Represents a struct field defined in the Delta table schema.

Enums

ApplyLogError

Error related to Delta log application

DeltaTableError

Delta Table specific error

DeltaTransactionError

Error returned by the DeltaTransaction struct

LoadCheckpointError

Error related to checkpoint loading

SchemaDataType

Enum with variants for each top level schema data type.

Functions

crate_version

Returns rust crate version, can be use used in language bindings to expose Rust core version

open_table

Creates and loads a DeltaTable from the given path with current metadata. Infers the storage backend to use from the scheme in the given table path.

open_table_with_ds

Creates a DeltaTable from the given path. Loads metadata from the version appropriate based on the given ISO-8601/RFC-3339 timestamp. Infers the storage backend to use from the scheme in the given table path.

open_table_with_version

Creates a DeltaTable from the given path and loads it with the metadata from the given version. Infers the storage backend to use from the scheme in the given table path.

Type Definitions

DeltaDataTypeInt

Type alias for i32/Delta int

DeltaDataTypeLong

Type alias for i64/Delta long

DeltaDataTypeTimestamp

Type alias representing the expected type (i64/ms since Unix epoch) of a Delta timestamp.

DeltaDataTypeVersion

Type alias representing the expected type (i64) of a Delta table version.

Guid

Type alias for a string expected to match a GUID/UUID format