Crate delta_kernel 

Delta Kernel

Delta-kernel-rs is an experimental Delta implementation focused on interoperability with a wide range of query engines. It supports table reads, table creation, appends, file removals, deletion-vector updates, and limited schema evolution. This library defines the traits that connectors implement to provide I/O and expression evaluation. The delta_kernel_default_engine crate provides a ready-to-use implementation based on Arrow, object_store, and Tokio.

A full rust example for reading table data using the default engine can be found in the read-table-single-threaded example (and for a more complex multi-threaded reader see the read-table-multi-threaded example). An example for reading the table changes for a table using the default engine can be found in the read-table-changes example. The write-table example demonstrates how to write data to a Delta table using the default engine.

Engine trait

The Engine trait allows connectors to bring their own implementation of functionality such as reading and writing Parquet and JSON files, listing files in storage, and evaluating expressions. It exposes handler traits for each capability. When adding cloud-storage functionality to an engine, such as writing JSON files, make sure to test it against S3, Azure, and GCS.

Expression handling

EvaluationHandler creates an ExpressionEvaluator for an Expression or a PredicateEvaluator for a Predicate. Each evaluator can process multiple data batches.

File system interactions

Delta Kernel needs to perform some basic operations against file systems like listing and reading files. These interactions are encapsulated in the StorageHandler trait. Implementers must take care that all assumptions on the behavior of the functions - like sorted results - are respected.

Reading log and data files

Delta Kernel requires the capability to read and write JSON and Parquet files, exposed via the JsonHandler and ParquetHandler respectively. Their read methods receive the context needed to execute the operation.

Re-exports

pub use crc::FileSizeHistogram;

pub use crc::FileStats;

pub use utils::try_parse_uri;internal-api

pub use utils::CollectInto;internal-api

pub use utils::FoldWithOption;internal-api

pub use cancellation::CancellationToken;

pub use cancellation::CancellationTokenRef;

pub use cancellation::CancelledFuture;

pub use engine_data::EngineData;

pub use engine_data::FilteredEngineData;

pub use engine_data::FilteredRowVisitor;

pub use engine_data::GetData;

pub use engine_data::RowIndexIterator;

pub use engine_data::RowVisitor;

pub use error::DeltaResult;

pub use error::DeltaResultIterator;

pub use error::DeltaResultIteratorStatic;

pub use error::Error;

pub use expressions::Expression;

pub use expressions::ExpressionRef;

pub use expressions::Predicate;

pub use expressions::PredicateRef;

pub use plans::IoOperation;declarative-plans

pub use plans::Operation;declarative-plans

pub use plans::PlanBuilder;declarative-plans

pub use plans::PlanExecutor;declarative-plans

pub use plans::PlanResult;declarative-plans

pub use snapshot::Snapshot;

pub use snapshot::SnapshotRef;

pub use delta_kernel_derive;

pub use arrow_59 as arrow;

pub use parquet_59 as parquet;

Modules

actions

Provides parsing and manipulation of the various actions defined in the Delta specification

cancellation

Cooperative cancellation for long-running Kernel reads.

checkpoint

This module implements the API for customized Delta checkpoint writes, where the caller drives the write themselves. The entry point is Snapshot::create_checkpoint_writer.

column_trieinternal-api

A trie (prefix tree) for efficient column path matching.

commit_range

Read a contiguous range of raw Delta commits.

committer

The committer module provides a Committer trait which allows different implementations to define how to commit transactions to a catalog or filesystem. For catalog-managed tables, a Committer specific to the managing catalog should be provided. For non-catalog-managed tables, the FileSystemCommitter should be used to commit directly to the object store (via put-if-absent call to storage to atomically write new commit files).

crcinternal-api

CRC (version checksum) file support.

enginearrow-conversion or declarative-plans or default-engine-base

Engine infrastructure shared by Engine implementations.

engine_data

Traits that engines need to implement in order to pass data between themselves and kernel.

error

Definitions of errors that the delta kernel can encounter

expressions

Definitions and functions to create and manipulate kernel expressions

history_manager

This module provides functions for performing timestamp queries over the Delta Log, translating between timestamps and Delta versions.

incremental_scan

Streams the file-action diff between two table versions. See IncrementalScanBuilder for the entry point.

kernel_predicates

Support for kernel-driven predicate evaluation via the KernelPredicateEvaluator trait. Various trait implementations are used for partition pruning, stats-based data skipping, and parquet row group filtering. The evaluation is normally performed over Scalar values, but data skipping “evaluation” actually produces a transformed predicate that replaces column references with stats column references, which log replay will instruct the engine to evaluate.

last_checkpoint_hintinternal-api

Utilities for reading the _last_checkpoint file. Maybe this file should instead go under log_segment module since it should only really be used there? as hint for listing?

log_replayinternal-api

This module provides log replay utilities.

log_segmentinternal-api

Represents a segment of a delta log. LogSegment wraps a set of checkpoint and commit files.

log_segment_filesinternal-api

LogSegmentFiles is a struct holding the result of listing the delta log. Currently, it exposes four APIs for listing:

metrics

Metrics collection for Delta Kernel operations.

object_storearrow-59 and (arrow-58 or arrow-59)

parallelinternal-api

Two-phase log replay for parallel execution of checkpoint processing.

partition

Partition utilities for Delta table writes.

pathinternal-api

Utilities to make working with directory and file paths easier

plansdeclarative-plans

Declarative plans: the kernel describes data work as a relational plan; the engine executes it.

reserved_field_ids

Reserved field IDs for metadata columns in Delta tables.

row_trackingtest-utils

scan

Functionality to create and execute scans (reads) over data stored in a delta table

schema

Definitions and functions to create and manipulate kernel schema

snapshot

In-memory representation of snapshots of tables (snapshot is a table at given point in time, it has schema etc.)

struct_patch

Struct patches: sparse, O(changes) edits to the fields of an input struct.

table_changes

Reads a table’s change data feed between two versions.

table_configuration

This module defines TableConfiguration, a high level api to check feature support and feature enablement for a table at a given version. This encapsulates Protocol, Metadata, Schema, TableProperties, and ColumnMappingMode. These structs in isolation should be considered raw and unvalidated if they are not a part of TableConfiguration. We unify these fields because they are deeply intertwined when dealing with table features. For example: To check that deletion vector writes are enabled, you must check both both the protocol’s reader/writer features, and ensure that the deletion vector table property is enabled in the TableProperties.

table_features

table_properties

Delta Table properties. Note this module implements per-table configuration which governs how table-level capabilities/properties are configured (turned on/off etc.). This is orthogonal to protocol-level ‘table features’ which enable or disable reader/writer features (which then usually must be enabled/configured by table properties).

transaction

transforms

Shared transform infrastructure.

utilsinternal-api

Various utility functions/macros used throughout the kernel

Macros

transform_output_type

Defines a transform’s Output and Residual associated types.

Structs

ActionReconciliationIterator

Iterator over action reconciliation data.

ActionReconciliationIteratorState

Stats for ActionReconciliationIterator

FileMeta

The metadata that describes an object.

LogCompactionWriter

Writer for log compaction files

LogPath

A path to a valid delta log file. You can parse a given FileMeta into a LogPath using LogPath::try_new.

ParquetFooter

Metadata from a Parquet file footer.

Traits

AsAny

Extension trait that makes it easier to work with traits objects that implement Any, implemented automatically for any type that satisfies Any, Send, and Sync. In particular, given some trait T: Any + Send + Sync, it allows upcasting T to dyn Any + Send + Sync, which in turn allows downcasting the result to a concrete type.

DynPartialEq

Extension trait that facilitates object-safe implementations of PartialEq.

Engine

The Engine trait encapsulates the functionality an engine or connector provides to operate on Delta tables.

EvaluationHandler

Provides expression evaluation capability to Delta Kernel.

EvaluationHandlerExtensioninternal-api

Internal trait to allow us to have a private create_one API that’s implemented for all EvaluationHandlers.

ExpressionEvaluator

Trait for implementing an Expression evaluator.

IntoEngineDatainternal-api

A trait that allows converting a type into (single-row) EngineData

JsonHandler

Provides JSON handling functionality to Delta Kernel.

ParquetHandler

Provides Parquet file related functionalities to Delta Kernel.

PredicateEvaluator

Trait for implementing a Predicate evaluator.

StorageHandler

Provides file system related functionalities to Delta Kernel.

Functions

should_compact

Determine if log compaction should be performed based on the commit version and compaction interval.

Type Aliases

FileDataReadResult

Data read from a Delta table file and the corresponding scan file information.

FileDataReadResultIterator

An iterator of data read from specified files

FileIndex

FileSize

FileSlice

A specification for a range of bytes to read from a file location

Version

Delta table version is 8 byte unsigned int