Crate datafusion_ducklake

Expand description

§DataFusion-DuckLake

A DataFusion extension that adds support for DuckLake, an integrated data lake and catalog format.

§Overview

DuckLake uses:

Catalog Database: SQL database (DuckDB, SQLite, PostgreSQL, MySQL) storing metadata as SQL tables
Data Storage: Apache Parquet files stored on disk/object storage

This extension provides read and write access to DuckLake catalogs through DataFusion’s catalog and table provider interfaces. Writes are feature-gated (write-sqlite, write-postgres); see COMPATIBILITY.md for the full backend/feature matrix.

§Example

use datafusion::prelude::*;
use datafusion_ducklake::{DuckLakeCatalog, SqliteMetadataProvider};

// Create a DataFusion session context
let ctx = SessionContext::new();

// Create a metadata provider for the catalog database. Each backend has its
// own provider behind its own feature; `metadata-sqlite` is on by default.
let provider = SqliteMetadataProvider::new("sqlite:path/to/catalog.sqlite").await?;

// Register a DuckLake catalog with the provider
let catalog = DuckLakeCatalog::new(provider)?;
ctx.register_catalog("ducklake", std::sync::Arc::new(catalog));

// Query tables from the catalog
let df = ctx.sql("SELECT * FROM ducklake.main.my_table").await?;
df.show().await?;

§Catalog I/O runs on a runtime of this crate’s own

DataFusion resolves catalog, schema and table names through synchronous trait methods, so a catalog query is awaited from a blocking context somewhere in here. That query is driven on a tokio runtime this crate owns rather than on the caller’s: two worker threads, started the first time a SQLite, PostgreSQL or MySQL catalog opens a pool or answers a call, shared by every provider and writer in the process, and never shut down. Behind them is a blocking pool capped at sixteen more, which carry the address resolution a pool does while it opens a connection. Tokio names both pools alike, so a process counting threads called ducklake-catalog can transiently see eighteen.

That is what lets a catalog call work wherever it is made from — a current_thread runtime, a multi-threaded one with every thread already busy, or a plain std::thread — and keeps the number of catalog calls in flight from being something an embedder has to size its own runtime against.

It covers the pools this crate opens: SqliteMetadataProvider::new, PostgresMetadataProvider::new, MySqlMetadataProvider::new and the metadata writers’ constructors all open theirs on that runtime. A pool adopted through from_pool or with_pool keeps whatever driver opened its connections, and a sqlx connection is only ever reported readable by that driver, so a catalog call through an adopted pool must not block the runtime the pool came from — under a current_thread runtime it blocks exactly that, and the call fails when the pool’s acquire timeout expires. Those constructors say what they do and do not move.

The cost is the threads this library starts without being asked, which is worth knowing if you are counting them.

Re-exports§

pub use catalog::DuckLakeCatalog;
pub use error::DuckLakeError;
pub use error::TypeChangeOperation;
pub use error::TypeChangeWriteMode;
pub use metadata_provider::ColumnTag;
pub use metadata_provider::DuckLakeFileData;
pub use metadata_provider::DuckLakeTableFile;
pub use metadata_provider::DuckLakeTag;
pub use metadata_provider::MetadataProvider;
pub use metadata_provider::ObjectTag;
pub use metadata_provider::SnapshotChangeMetadata;
pub use metadata_provider::TagObjectType;
pub use metadata_provider::TagTarget;
pub use partition::PartitionSpec;
pub use partition::PartitionSpecColumn;
pub use partition::PartitionTransform;
pub use schema::DuckLakeSchema;
pub use sort::NullOrder;
pub use sort::SortDirection;
pub use sort::SortField;
pub use sort::SortSpec;
pub use table::DuckLakeTable;
pub use table_functions::register_ducklake_functions;
pub use metadata_provider_sqlite::SqliteMetadataProvider;

Modules§

catalog: DuckLake catalog provider implementation
column_rename: Custom execution plan for renaming columns
delete_filter: Custom execution plan for filtering deleted rows of one data file
encryption: Encryption support for reading encrypted Parquet files in DuckLake.
error: Error types for the DuckLake DataFusion extension
information_schema: Information schema implementation for DuckLake catalog metadata
inlined_filter: Conservative filter translation for metadata-catalog inlined scans.
metadata_provider
metadata_provider_sqlite: SQLite metadata provider for DuckLake catalogs.
partition: DuckLake table partitioning: transform model + partition spec.
path_resolver: Path resolution utilities for DuckLake
row_id: Physical row position and synthetic rowid column injection for DuckLake row lineage.
schema: DuckLake schema provider implementation
snapshot_filter: Per-row time-travel visibility filter for DuckLake partial data files.
sort: DuckLake table sort order: sort spec model.
stats_encode: DuckDB-canonical value → VARCHAR encoding for DuckLake column statistics.
stats_filter: Lower a pushed-down predicate to DuckLake catalog-statistics SQL.
table: DuckLake table provider implementation
table_changes: Table changes (CDC) functionality for DuckLake
table_deletions: Table deletions functionality for DuckLake
table_functions: User-Defined Table Functions (UDTFs) for DuckLake catalog metadata
types: Type mapping from DuckLake types to Arrow types

Functions§

register_snapshot_consistency: Add the snapshot consistency analyzer rule to a session. Without it, each table reference of a statement reads the latest snapshot at its own lookup.

Type Aliases§

Result