Crate delta_kernel

Delta Kernel

Delta-kernel-rs is an experimental Delta implementation focused on interoperability with a wide range of query engines. It currently only supports reads. This library defines a number of traits which must be implemented to provide a working “delta reader”. The are detailed below. There is a provided “default engine” that implenents all these traits and can be used to ease integration work. See DefaultEngine for more information.

A full rust example for reading table data using the default engine can be found here

Engine traits

The Engine trait allow connectors to bring their own implementation of functionality such as reading parquet files, listing files in a file system, parsing a JSON string etc. This trait exposes methods to get sub-engines which expose the core functionalities customizable by connectors.

Expression handling

Expression handling is done via the ExpressionHandler, which in turn allows the creation of ExpressionEvaluators. These evaluators are created for a specific predicate Expression and allow evaluation of that predicate for a specific batches of data.

File system interactions

Delta Kernel needs to perform some basic operations against file systems like listing and reading files. These interactions are encapsulated in the FileSystemClient trait. Implementors must take take care that all assumptions on the behavior if the functions - like sorted results - are respected.

Reading log and data files

Delta Kernel requires the capability to read json and parquet files, which is exposed via the JsonHandler and ParquetHandler respectively. When reading files, connectors are asked to provide the context information it requires to execute the actual read. This is done by invoking methods on the FileSystemClient trait.

Re-exports

• pub use engine_data::DataVisitor;
• pub use engine_data::EngineData;
• pub use error::DeltaResult;
• pub use error::Error;
• pub use expressions::Expression;
• pub use table::Table;

Modules

• actions
  Provides parsing and manipulation of the various actions defined in the Delta specification
• column_mapping
  Code to handle column mapping, including modes and schema transforms
• enginedefault-engine or sync-engine or arrow-conversion
  Provides engine implementation that implement the required traits. These engines can optionally be built into the kernel by setting the default-engine or sync-engine feature flags. See the related modules for more information.
• engine_data
  Traits that engines need to implement in order to pass data between themselves and kernel.
• error
  Defintions of errors that the delta kernel can encounter
• expressions
  Definitions and functions to create and manipulate kernel expressions
• scan
  Functionality to create and execute scans (reads) over data stored in a delta table
• schema
  Definitions and functions to create and manipulate kernel schema
• snapshot
  In-memory representation of snapshots of tables (snapshot is a table at given point in time, it has schema etc.)
• table
  In-memory representation of a Delta table, which acts as an immutable root entity for reading the different versions
• transaction

Structs

• FileMeta
  The metadata that describes an object.

Traits

• Engine
  The Engine trait encapsulates all the functionality an engine or connector needs to provide to the Delta Kernel in order to read the Delta table.
• ExpressionEvaluator
  Trait for implementing an Expression evaluator.
• ExpressionHandler
  Provides expression evaluation capability to Delta Kernel.
• FileSystemClient
  Provides file system related functionalities to Delta Kernel.
• JsonHandler
  Provides JSON handling functionality to Delta Kernel.
• ParquetHandler
  Provides Parquet file related functionalities to Delta Kernel.

Type Aliases

• FileDataReadResult
  Data read from a Delta table file and the corresponding scan file information.
• FileDataReadResultIterator
  An iterator of data read from specified files
• FileSlice
  A specification for a range of bytes to read from a file location
• Version
  Delta table version is 8 byte unsigned int