Crate moraine

Expand description

Moraine brings a SlateDB backend to DuckLake: a DuckLake catalog implemented on a transactional KV store over object storage, instead of the usual relational catalog database.

Exactly one process holds a read-write Catalog per store at a time (opening a second fences the first); any number of processes may read snapshots concurrently. Schemas, tables, views, data files, and statistics all commit through the same transaction; catalog options live outside the snapshot protocol (last-write-wins, no snapshot minted).

§A worked example

Open a catalog on a store, evolve its schema through commits, and read both the current state and any past snapshot:

let catalog = Catalog::open(Arc::new(InMemory::new()), CatalogOptions::default()).await?;

let v1 = catalog
    .commit(|tx| {
        let sales = tx.create_schema("sales")?;
        tx.create_table(
            sales,
            "orders",
            &[ColumnDef {
                name: "id".into(),
                column_type: "BIGINT".into(),
                nulls_allowed: false,
                default_value: None,
                children: Vec::new(),
            }],
        )?;
        Ok(())
    })
    .await?;

// The head sees the latest shape (plus the bootstrap-minted `main`)...
let head = catalog.snapshot().await?;
assert_eq!(head.schemas().len(), 2);

// ...while `v1` remains queryable by time travel, forever.
let past = catalog.snapshot_at(v1).await?;
assert_eq!(past.schemas().len(), 2);

See examples/walkthrough.rs for a longer tour that also alters and renames a table across a second commit. Data files register through the same Catalog::commit path, minting a snapshot without bumping the schema version.

§Inlined data

A small insert need not cost a Parquet file. Transaction::inline_insert stores a chunk of rows in the catalog itself — Arrow IPC bytes the caller encodes, carried by the same single write the commit already performs — and the rows draw ids from the table’s row-id counter exactly as a registered file’s do. ReadOnlyCatalog::recent_rows reads them back — or ReadOnlyCatalog::recent_rows_at, for the rows a past snapshot saw — Transaction::inline_delete tombstones one, and Transaction::flush_inlined_data drains them into a data file the caller wrote, preserving their ids and backdating the file’s record so time travel across the flush still finds them exactly once.

§Readers

Catalog::open_read_only opens the same store without becoming its writer, so it never fences one. What it hands back is a ReadOnlyCatalog, which carries the reads and no mutator at all — so committing through a read-only handle is a compile error rather than a runtime one, and a Catalog serves the same reads by dereferencing to it. Read-only comes in two forms, and the difference is what the reader’s credentials must be allowed to do.

The default follows the latest state: it sees every commit as it lands, and pays for that by writing a checkpoint into the manifest on open and refreshing it while it lives. The other form pins the reader to a checkpoint minted in advance, and writes nothing whatsoever — for a deployment whose readers hold strictly read-only credentials:

catalog.commit(|tx| tx.create_schema("sales").map(|_| ())).await?;
let checkpoint = catalog.create_checkpoint(None).await?;

let mut options = CatalogOptions::default();
options.checkpoint = Some(checkpoint);
let reader = Catalog::open_read_only(object_store, options).await?;
assert!(reader.snapshot().await?.schema_by_name("sales").is_some());

A pinned reader reads that cut and no other, however long it stays open — a commit made after the checkpoint is invisible to it. The checkpoint also pins the objects it references against SlateDB’s own collection until it expires or Catalog::delete_checkpoint removes it, so give one a lifetime unless something else will.

§Maintenance

Dropping an equality index (or a table that has one) makes its entries invisible immediately but does not delete them, so the range would leak without a sweep. Catalog::maintain reclaims those ranges:

let catalog = Catalog::open(Arc::new(InMemory::new()), CatalogOptions::default()).await?;

let report = catalog.maintain(MaintenanceRequest::default()).await?;
// Nothing has been dropped, so there is nothing to reclaim.
assert_eq!(report.indexes_swept, 0);

A pass mints no snapshot and leaves head unchanged, and deletes in bounded batches (MaintenanceRequest::batch_size) so a large reclamation never holds the writer. It is safe to interrupt: an abandoned pass leaves a smaller one for next time, never a torn one. Deciding what is dead rests on catalog ids never being reused, so a sweep can run alongside a live writer without coordinating with it.

Beneath the catalog, the store itself accumulates. Every key lives in a subspace, each subspace is its own tree, and overwriting a key leaves the old version readable-through until a merge rewrites the SSTs holding it — which the substrate’s own scheduler only does under write pressure, so a store that goes quiet keeps its dead weight indefinitely. ReadOnlyCatalog::store_census says where the weight is, from the store’s manifest alone:

let catalog = Catalog::open(Arc::new(InMemory::new()), CatalogOptions::default()).await?;
catalog.commit(|tx| tx.create_schema("sales").map(|_| ())).await?;

let census = catalog.store_census(CensusRequest::default()).await?;
// Every subspace is reported, so two censuses compare row by row.
assert!(census.subspaces.iter().any(|s| s.subspace == SubspaceName::Current));

// ...and the merge collapses a subspace's sorted runs into one.
let report = catalog.compact_store(CompactStoreRequest::default()).await?;
assert!(!report.merges.is_empty());

The census costs a manifest read plus one listing and is available read-only; its per-subspace figures count SSTs that have been written out, so a commit still in the write-ahead log is in none of them. The listing is what catches the rest: StoreCensus::unaccounted_bytes reports what the object store holds outside those SSTs, and a large figure there means a slow attach no merge will fix — an unpinned reader replays the log before it materializes anything. Ask for CensusRequest::count_live_entries and it also scans, which costs a full read of the store and is the only way to learn what fraction of a subspace is live. Catalog::compact_store then merges: moraine picks no sources and no destination, and the plan the substrate makes for a whole tree is what permits dropping a tombstone rather than carrying it forward. It needs the writer, since the compactor that executes a merge runs inside it.

Snapshot expiry, file cleanup, and data-file compaction all belong to DuckLake and run through its own functions; the SlateDB store collects its superseded objects itself, unprompted. A host embedding this crate calls these verbs on whatever cadence it likes — the crate spawns no threads and schedules nothing. Under the DuckDB extension, all of it is sequenced for you; see that crate’s docs for moraine_maintenance, moraine_store_census, and moraine_compact_store.

§Format migration

A store records the structural layout version it was written at. A binary opens any version it understands and refuses one it does not, rather than misreading it. Most version bumps are additive — a new subspace, no existing key moved — and need no rewrite: the store is stamped forward the first time it uses the newer feature, and older stores stay readable.

A bump that moves existing keys does need a rewrite, and Catalog::migrate performs it: a one-way, crash-resumable pass that takes the single writer, walks the keyspace under a durable cursor, and flips the version stamp in one final atomic batch. It is a separate verb from opening a catalog, so the operator chooses when to pay for it. While it runs, every reader refuses the store.

No such rewrite exists yet, so migrate reports a no-op against every store this release can open.

§Features

fault-injection — compiles the crash-injection seams the migration driver consults between its durable batches, and exposes CrashPoint, inject_crash, SyntheticMigration, install_migration, and CrashCase so a test can install a migration unit and stop it at a named durable boundary. Off by default; a build without it carries no fault surface.
fuzzing — exposes the codec and read-path decode entry points the fuzz/ targets drive. They live in their own crate and so cannot reach the crate-private codecs; nothing else needs this. Off by default.

§Diagnostics

The crate emits tracing events and installs no subscriber. Rare, consequential moments log at info (an open, the once-ever bootstrap, a format-stamp upgrade) or warn (a spent retry budget, an oversized commit refused); per-commit summaries and retry traces log at debug. Every event is per-operation — never per row, entry, or file. An embedding host sees them by installing any subscriber. The DuckDB extension consumes its own — it cannot share the host’s — and forwards them to duckdb_logs; see that crate’s logging module.

§Layering

catalog — the DuckLake domain model. Never touches SlateDB directly.
store — the SlateDB layer: key layout and value codecs. Knows nothing about DuckLake semantics.
transaction — the commit protocol turning a catalog transaction into an atomic store write.

Structs§

BuildStep: How much one step of a staged index build may commit. A step ends at whichever bound it reaches first, and always carries at least one entry. entries bounds one step’s entry buffer; derivation holds the step being filled and the one committing, plus a fixed window of decoded source batches read ahead. bytes bounds the single object-store request a batch becomes.
CacheIdentity: The object namespace shared by cache users. Equal identities assert that the same object path names the same immutable contents.
CacheStatus: Current size and pressure of the process-shared caches. The disk figure is a configured capacity, not live occupancy.
CacheTally: What the cache has served, by tier. cache_tally reports the process’s; ReadOnlyCatalog::cache_tally one attach’s.
Catalog: A read-write handle to a moraine catalog: cheap to clone, drives reads and commits.
CatalogOptions: Options for opening a catalog.
CatalogSnapshot: An immutable catalog view at one snapshot.
CensusRequest: How a ReadOnlyCatalog::store_census call should run.
ColumnAlteration: A change to one column. None fields leave the attribute untouched; default_value uses a nested Option so “clear the default” (Some(None)) is distinct from “leave it” (None).
ColumnDef: A column definition: the input to table creation and column addition.
ColumnId: Identifies a column within its table (a field id: stable across renames and never reused, even after the column is dropped).
ColumnInfo: A live column: its definition plus identity and position.
ColumnOrder: The sort order of one indexed column: its direction and NULL placement. Defaults to ascending, NULLS LAST.
ColumnStats: A column’s table-level statistics. Min/max are opaque strings, stored verbatim — moraine never interprets or merges them.
CompactStoreReport: What a store merge did, subspace by subspace.
CompactStoreRequest: How a Catalog::compact_store call should run.
DataFile: A data file to register: the file already exists on object storage (data before metadata). row_id_start is allocated by the commit, never caller-provided.
DataFileId: Identifies a data file.
DataFileInfo: A live data file, as read from a snapshot.
DataStore: An object store holding data files, named once for the caches that key on it. Build one per store and clone it to share cached reads.
DeleteFile: A delete file to register, targeting one data file’s rows.
DeleteFileId: Identifies a delete file.
DeleteFileInfo: A live delete file, as read from a snapshot.
DeleteFileRegistration: One delete file to land through Catalog::commit_located_deletion: the caller has already written it — the union of an existing delete file’s positions (if any) with newly located deletions — and supplies enough to register it in place of the file it replaces.
ExcludedPositions: File positions a caller treats as deleted on top of the snapshot’s own delete files, such as a transaction’s uncommitted deletions.
ExistingDeleteFile: A delete file already registered against a data file a caller is about to delete more rows from.
FileColumnStats: Per-column statistics of one data file. Min/max are opaque strings, stored verbatim — moraine never interprets or merges them.
FileIndexEntry: One writer-supplied index entry for a row of a registered data file. The ordinal is the row’s 0-based position in the file; the commit maps it to a row id (row_id_start + ordinal).
FileIndexRemoval: One writer-supplied entry removal for a registered delete file: the killed row’s id and the values it was indexed under. Against a dense-range target the id must lie inside the target’s row-id range; against a per-row-id target it is the file’s embedded id, supplied verbatim.
FileRowCandidate: Where one stable row id may physically live at head.
FlushedDataFile: A data file a flush wrote, carrying rows that were inlined before it. Unlike an ordinary registration, the rows keep the ids they were inlined under and the record is backdated, so a pre-flush time-travel read finds them in the file rather than in the drained chunks.
IndexDef: The definition of an equality index to create.
IndexEntry: One writer-supplied index entry: a row and its indexed column values, in the index’s column order. A None in any position is SQL NULL; a row with any NULL indexed value is stored as a collision-exempt multi-shaped entry, so IS NULL finds it and a unique index still admits unlimited NULL rows.
IndexId: Identifies an equality index within its table (allocated from the global catalog-id counter).
IndexInfo: A live equality index, as read from a snapshot.
IndexReadIdentity: An opaque read revision, separated from the resources pinning it.
IndexReadScope: Index, file, and inline reads pinned to one committed store revision. Dropping the scope and its read clones releases the pinned transaction.
InlineChunk: A chunk of rows to inline: one Arrow IPC record-batch body over the table’s user columns, decoded against the schema-only stream recorded for its schema_version. Row ids are allocated by the commit from the table’s row-id counter, never caller-provided — the same allocation a data-file registration performs.
LiveCount: Live keys of one subspace, and the bytes they occupy.
LocatedDeletion: One requested row’s exact position within one of a table’s current data files, with the delete file currently registered against it, if any.
LocatedPositions: What ReadOnlyCatalog::locate_row_positions resolved: one LocatedDeletion per data file carrying a requested row, the row ids resolved as live inlined rows, and — when any position was resolved — the directory a new delete file for this table belongs in.
LocatedRowScan: A selective reader emitting one projected Arrow batch at a time. Dropping the reader cancels its remaining reads.
MacroId: Identifies a macro.
MacroImplementationDef: One implementation (arity overload) of a macro.
MacroInfo: A macro with its implementations.
MacroParameterDef: One parameter of a macro implementation. An absent default stores default_value: None with default_value_type "unknown", matching the row DuckLake writes.
MaintenanceReport: What a maintenance pass reclaimed.
MaintenanceRequest: What a maintenance pass should reclaim.
MaintenanceStatusPass: One completed maintenance pass in the catalog’s durable status history.
MaintenanceStatusStep: One step reported by a completed maintenance pass.
MappingId: Identifies a column mapping (allocated by DuckLake from the same counter as data-file ids).
MappingInfo: A column mapping for externally written Parquet: immutable once created, referenced by data files via their mapping_id.
MemoryTally: Logical memory Moraine can attribute without sampling process RSS. Cache occupancy is process-wide; the other fields belong to one catalog.
MigrationReport: What one migrate call did.
MigrationRequest: How a Catalog::migrate call should run.
NameMappingDef: One ducklake_name_mapping row: how one physical column of an externally written file resolves to a table field. column_id is a 0-based ordinal local to the mapping; parent_column, when present, references a smaller ordinal (parents precede children in preorder).
ObjectStoreTally: Physical object-store requests SlateDB has issued for one catalog, retries included. WAL objects land in the main_* fields while the WAL shares the main store. Durations sum completed request latency, so concurrent requests can exceed wall-clock time.
PartitionColumnDef: One key of a partition spec: a source column and the transform applied to it. Transforms are stored verbatim as DuckLake writes them (identity, year, bucket(16), …); moraine never parses or applies one. A bare partition column carries identity rather than an empty transform, so every key is explicit.
PartitionId: Identifies a partition spec within its table (allocated from the global catalog-id counter).
PartitionSpec: A table’s partition spec: its partition keys in order. A table has at most one live spec; setting a new one ends the old, and files written under the old spec keep referencing it.
ReadOnlyCatalog: The read surface of a moraine catalog: cheap to clone, drives every read. This is what a read-only attach hands back, and what a read-write Catalog derefs to.
RecentRow: One inlined row, with the Arrow IPC bytes a caller needs to decode it. Rows of one chunk share one chunk_body, and rows of one schema version share one arrow_schema.
RowSummaryOccupancy: Resident file row-id summaries by shape, and their bytes together; the summary share of auxiliary_metadata_occupancy_bytes.
RowSummaryWarmth: What one warming pass did.
ScheduledDeletion: One ducklake_files_scheduled_for_deletion row: a path awaiting physical deletion, decoupled from the expiry that scheduled it.
SchemaId: Identifies a schema.
SchemaInfo: A live schema.
SnapshotId: Identifies a committed snapshot.
SnapshotInfo: Identity and metadata of a committed snapshot.
SortId: Identifies a sort spec within its table (allocated from the global catalog-id counter).
SortKeyDef: One key of a sort spec: a SQL expression and how it orders. Every field is stored verbatim as DuckLake writes it — the expression and its dialect, the sort_direction (ASC/DESC), the null_order (NULLS_FIRST/NULLS_LAST) — and moraine parses or evaluates none of them.
SortSpec: A table’s sort spec: its sort keys in order. A table has at most one live spec; setting a new one ends the old. Unlike a partition spec, no data file records the sort spec it was written under — a sort spec is a write-time instruction, not file provenance.
StoreCensus: What a store weighs, subspace by subspace.
StoreObjects: What the object store holds, by the kind of object holding it. The per-subspace figures account for SST bytes only; the write-ahead log shows up only here.
SubspaceCensus: What one subspace weighs. The physical figures come from the store’s manifest and count what has been written out, superseded versions and tombstones included; a write still in the write-ahead log is in none of them.
SubspaceMerge: What one subspace’s merge did.
TableId: Identifies a table.
TableInfo: A live table.
TableStats: A table’s statistics.
TagEntry: One tag on a catalog object: a key/value row, begin/end-versioned like any temporal row. Ended entries stay readable for time travel until garbage-collected.
Timestamp: An instant, as microseconds from the Unix epoch in UTC — the precision the catalog persists and the extension ABI carries. Negative counts precede the epoch.
Transaction: The mutation handle a commit closure receives.
ViewId: Identifies a view.
ViewInfo: A live view.

Enums§

CachePreload: How much of a store to load into the on-disk object cache as it opens.
CompactionTarget: Which trees a Catalog::compact_store call merges.
Direction: Sort direction of an indexed column.
Error: Errors returned by moraine operations.
IndexKeyValue: A single non-null indexed column value, typed by the column’s category. NULL is not a variant here — it is the None of a column’s optional value, which encodes to a leading null flag.
IndexMaintenance: When SQL writes add entries to an index.
IndexState: The build lifecycle of an equality index.
IntWidth: Byte width of a fixed-width integer column.
MergeOutcome: How one subspace’s merge ended.
NullOrder: Where NULLs sort relative to the non-null values of an indexed column, independent of the column’s Direction.
OptionScope: An option scope: global, or one schema or table.
SubspaceName: One subspace of the keyspace, as a census names it. A segment this build does not recognize is reported as Unknown rather than dropped.
TagTarget: A catalog object a tag can be attached to. Schemas, tables, and views share one id space, so the variant selects which entity the verb validates against and how the change is classified, not how the tag is keyed. Column tags are carried on the column itself and are not reachable here.

Functions§

cache_status: Current process-wide cache sizing and occupancy, summed over every store’s cache. The metadata figure is the ceiling past which metadata stops being protected, not a partition; the block figure is the whole budget.
cache_tally: What the process’s block cache has served since it was built, metadata and data blocks counted apart. Process-wide, outliving the catalogs that produced it; use it to size CatalogOptions::cache_memory and cache_size, and ReadOnlyCatalog::cache_tally to see which attach is spending them.

Type Aliases§

CommitMember: One member of a Catalog::commit_group batch: a closure authoring one logical commit.
Result: Crate-wide result alias.

Crate moraine

Crate moraine Copy item path

§A worked example

§Inlined data

§Readers

§Maintenance

§Format migration

§Features

§Diagnostics

§Layering

Structs§

Enums§

Functions§

Type Aliases§

Crate moraine