Expand description
The deltalake crate is currently just a meta-package shim for deltalake-core Delta Kernel module
The Kernel module contains all the logic for reading and processing the Delta Lake transaction log.
Modules§
- arrow
- Conversions between Delta and Arrow data types
- engine
- Provides an engine implementation that implements the required traits. The engine can optionally
be built into the kernel by setting the
default-enginefeature flag. See the related module for more information. - error
- Error types for Delta Lake operations.
- models
- Actions are the fundamental unit of work in Delta Lake. Each action performs a single atomic
operation on the state of a Delta table. Actions are stored in the
_delta_logdirectory of a Delta table in JSON format. The log is a time series of actions that represent all the changes made to a table. - scalars
- Auxiliary methods for dealing with kernel scalars
- schema
- transaction
- Add a commit entry to the Delta Table. This module provides a unified interface for modifying commit behavior and attributes
Structs§
- Add
- Defines an add action
- AddCDC
File - Delta AddCDCFile action that describes a parquet CDC data file.
- Array
Type - Checkpoint
Metadata - This action is only allowed in checkpoints following V2 spec. It describes the details about the checkpoint.
- Commit
Info - The commitInfo is a fairly flexible action within the delta specification, where arbitrary data can be stored. However the reference implementation as well as delta-rs store useful information that may for instance allow us to be more permissive in commit conflict resolution.
- Decimal
Type - Deletion
Vector Descriptor - Defines a deletion vector
- Domain
Metadata - The domain metadata action contains a configuration (string) for a named metadata domain
- Eager
Snapshot - A snapshot of a Delta table that has been eagerly loaded into memory.
- Invariant
- An invariant for a column that is enforced on all writes to a Delta table.
- LogData
Handler - Provides semanitc access to the log data.
- Logical
File View - Provides semantic, typed access to file metadata from Delta log replay.
- MapType
- Metadata
- Protocol
- Remove
- Represents a tombstone (deleted file) in the Delta log.
- Scan
- Scan
Builder - Builder to scan a snapshot of a table.
- Sidecar
- The sidecar action references a sidecar file which provides some of the checkpoint’s file actions. This action is only allowed in checkpoints following V2 spec.
- Snapshot
- A snapshot of a Delta table
- Struct
Field - Struct
Type - A struct is used to represent both the top-level schema of the table as well as struct columns that contain nested columns.
- Tombstone
View - Transaction
- Action used by streaming systems to track progress using application-specific versions to enable idempotency.
Enums§
- Action
- Column
Metadata Key - Data
Type - Error
- Isolation
Level - The isolation level applied during transaction
- Metadata
Value - Primitive
Type - Storage
Type - Storage type of deletion vector
- Table
Features - High level table features
Traits§
- Data
Check - A trait for all kernel types that are used as part of data checking
- Metadata
Ext - Extension trait for Metadata action
- Record
Batch Stream - Trait for types that stream RecordBatch
- Struct
Data Ext - Extension trait for Kernel’s
StructData. - Struct
Type Ext - Trait to add convenience functions to struct type
Functions§
- cast_
record_ batch - Cast recordbatch to a new target_schema, by casting each column array
- contains_
timestampntz - checks if table contains timestamp_ntz in any field including nested fields.
- new_
metadata - Please don’t use, this API will be leaving shortly!
Type Aliases§
- Delta
Result - A specialized
Resulttype for Delta Lake operations. - Schema
- Type alias for a top level schema
- Schema
Ref - Schema reference type
- SendableRB
Stream - Sendable
Record Batch Stream - Trait for a
StreamofRecordBatches that can be passed between threads - Sendable
Scan Metadata Stream