skade

Skaði, the winter queen — she keeps the icebergs in order.

This repo is Skade: a fast, pure-Rust thin layer over iceberg-rust, backed by a ridiculously fast skade-katalog.

• skade — the data plane: Arrow RecordBatch in, SQL out; gatling multi-core ingest, zero-copy recast, identity-partitioned + compressed writes, V3 tables. (skade/)
• skade-katalog — the catalog: pure-Rust, redb-backed, single-file ACID iceberg::Catalog. No SQL, no C deps.

skade-katalog

A pure-Rust iceberg::Catalog with cross-table transactions. A lock-free static search tree (Ragnar STree64) out front, continuously regenerated from a redb ACID backend. In-process — no network hop, no JVM, no JSON round-trip.

• table_exists — 2.0M ops/s, ~492× Nessie · ~677× Polaris (0.31 µs p50). Every core reads the front lock-free.
• Ingest saturates the box — 100% CPU, fully multicore. All cores encode Parquet in parallel; a gatling no-barrier pool keeps the writes overlapping, even over S3.
• ACID. Every mutation is one redb WriteTransaction. Atomic multi-table commits via RedbCatalog::atomic_release.

Full leaderboard (skade-katalog vs Nessie vs Polaris, every storage variant, all four capabilities) → Benchmarks.

Status

The Iceberg Catalog trait is fully implemented and tested against iceberg = "0.9.1". One gap: schema evolution (the upstream Transaction actions aren't public until iceberg-rust 0.10). See Known shortcomings.

skade — the data-plane companion crate

This repo also ships skade ("winter queen" — Skaði): fast Iceberg table writing/reading + ergonomic DataFusion SQL on top of RedbCatalog. One directory = catalog (catalog.redb) + warehouse; Arrow RecordBatch in, SQL out (wh.sql("SELECT … FROM a JOIN b …")). It packages the writer stack, read-to-Arrow primitive, schema bridges (incl. the unsigned widen/unwiden reinterprets znippy needs), and the windowed-Parquet bulk-ingest helpers that were proven in bench/. Like bench/, it is a detached cargo workspace (skade/), so this crate's manifest and lockfile stay untouched. See skade/README.md.

Read & write paths

Every call resolves left-to-right through these layers, falling through only on a miss. Latest-state reads ride the L1 pointer mirror; time-travel (snapshot-id) reads ride the Ragnar STree64 index. redb is always the source of truth behind them — the in-memory layers are keyed by immutable, content-addressed identifiers, so they can only be evicted, never stale.

The same routing in full — including the time-travel, write, and redb-direct paths the diagram leaves out:

LATEST READS
  table_exists(id) ───────► L1 mirror ──hit──► true
                                └─miss─► redb `tables` ─────► bool

  resolve_metadata(id) ───► L1 mirror ──► loc ──► L0 cache ──hit──► Arc<TableMetadata>
       (skips Table::build)     └─miss─► redb         └─miss─► FileIO read + parse JSON

  load_table(id) ─────────► L1 mirror ──► loc ──► L1.5 cache ──hit──► Table  (~100 ns clone)
                                └─miss─► redb          └─miss─► L0 cache ──► Table::build() + insert

TIME-TRAVEL READS  (by snapshot_id: i64)
  load_table_at / resolve_metadata_at(id, sid)
        ──► Ragnar STree64 ──hit──► loc ──► L1.5 / L0 ──► Table / Arc<TableMetadata>
                 └─miss (sid above cutoff)─► redb `commits` live tail ──► loc ──► …

  resolve_many(id, [sid; N])
        ──► Ragnar batch probe (1 pipelined pass) ──► redb `commits` (1 txn, misses only) ──► L0 per loc

WRITES  (create · register · update · drop · rename)
  …──► FileIO write …/<uuid>.metadata.json ──► redb WriteTransaction { tables CAS · commits log · meta++ }
        ──commit──► L1 mirror write-through (insert / remove) ──► maybe rebuild Ragnar (background)

  update_table ──► group-commit: N concurrent commits coalesce into 1 redb txn / 1 fsync   ← commit-burst lever

REDB-DIRECT  (no cache layer)
  list_namespaces · get_namespace · namespace_exists · list_tables ──► redb read txn (range scan)
  create / update / drop_namespace ──────────────────────────────────► redb WriteTransaction

Benchmarks

All numbers below are auto-filled by nornir docs render from the latest nornir bench run (machine/cores/version in each header). Don't hand-edit inside the generated regions — re-run the bench instead. Reproduce with the harness in bench/ (cargo run --bin bench-containers -- up in bench/, then nornir bench run skade-katalog from workspace_skade/).

Catalog read RPC — table_exists (storage-free, runs on every backend)

table_exists is a pure catalog RPC (no object storage), the apples-to-apples comparable; Nessie/Polaris go through the same iceberg::Catalog REST client.

v0.4.11 · oden · 32 cores · 2026-06-12

Embedded read fast paths (nornir-only)

resolve_metadata skips Table::build() (the lock-free L1+L0 floor); load_table is the full Catalog trait path after the L1.5 handle cache.

v0.4.11 · oden · 32 cores · 2026-06-12

Data plane — single-writer / many-processor ingest

All cores encode Parquet; one writer streams + commits (data-pipe). Backend- agnostic and storage-bound (shared client-side Parquet + S3), so rows/s converge across backends — this lifts all boats, it is not a catalog lever. Nessie runs the same shared RustFS S3 warehouse. Scan verifies the round-trip. The nornir local-FS row runs two file destinations — _nvme (PCIe-4.0 NVMe) and _ram (/dev/shm tmpfs) — to expose the pure storage floor.

v0.4.11 · oden · 32 cores · 2026-06-12

Commit-bursty — the catalog lever (group-commit)

Many small concurrent metadata commits — where skade-katalog coalesces commits into one redb txn/fsync. This is where the embedded catalog genuinely pulls ahead of a REST/JVM catalog (throughput and tail latency).

v0.4.11 · oden · 32 cores · 2026-06-12

TPC-H — analytical SQL (all 22 queries via DataFusion)

The full 8-table TPC-H schema loaded into Iceberg tables, queried through DataFusion (iceberg-datafusion) — per-query latency for all 22 canonical queries over the catalog's scan/manifest path. Authentic tpchgen data; TPCH_SF sets the scale (small by default). Answer values aren't validated here (that needs SF=1) — this measures plan + scan + execute latency end-to-end.

v0.4.11 · oden · 32 cores · 2026-06-12

TPC-H large warehouse (opt-in, big iron)

A full-scale run: ingest the 8-table warehouse at a large scale factor across all cores (partitioned parallel ingest), then run all 22 queries over it — reporting build vs query time, ingest rate, and the slowest query. Runs by default at a moderate scale; TPCH_WAREHOUSE_SF tunes it (default 10 ≈ 60 M lineitem rows / a few min; 100-200 for a 10-60 min big-iron workout), TPCH_WAREHOUSE=0 disables it, TPCH_WAREHOUSE_PARTS sets ingest partitions (default = all cores).

v0.4.11 · oden · 32 cores · 2026-06-12

TPC-H across catalogs × storage (skade-katalog vs Nessie vs Polaris)

The same SF warehouse built and all 22 queries run over each (catalog × storage) combination through DataFusion — the analytical-SQL path compared apples-to-apples (not just the table_exists RPC). The matrix covers every combo that's possible: nornir on file-NVMe, file-RAM, and S3; Nessie S3-only (it rejects a local file warehouse); Polaris on file (its S3 credential vending 301s against the RustFS endpoint — see bench/src/containers.rs). The REST targets need their container + the shared RustFS S3 warehouse (bench-containers up all), else they skip. TPCH_COMPARE_SF sets the scale (default 1.0 ≈ 8.66M rows).

Parallel scan patch: stock iceberg-datafusion 0.9.1 scans a table as a single DataFusion partition (one serial Parquet-decode stream), so on a 32-core box only ~1 core works and queries are ~7-8× slower than they should be. The repo vendors a patched iceberg-datafusion (bench/vendor/, relative-path [patch] used by both bench/ and skade/) that exposes one partition per file, so DataFusion decodes files across all cores. SF100 FILE query dropped ~1660s → ~197s (8.4×). These numbers reflect that patch; published skade builds get stock 0.9.1 until the patch is upstreamed.

At SF100 (866M rows) both S3-backed paths (skade_s3, nessie_s3) fail with hyper: connection closed before message completed finishing the Parquet writer — an S3 write-robustness limit under sustained heavy PUT load (rust-s3 / RustFS), not a catalog issue; the FILE-backed paths complete cleanly.

v0.4.11 · oden · 32 cores · 2026-06-12

OSM GeoParquet ingest (data plane)

End-to-end ingest of authentic OSM data: a nodes.parquet produced by the katana-osm / osm2geoparquet converter (OSM .osm/.bz2/.pbf → GeoParquet) is read, its schema mapped to Iceberg, and ingested into a fresh RedbCatalog table via the single-writer / many-processor pipeline, then scanned back to verify. NVMe vs RAM destinations isolate storage cost. Point OSM_GEOPARQUET at the file (OSM_MAX_ROWS caps rows); the osm_ingest_* benchers record a failure if it's unset, like the container-backed targets.

(no bench results)

ZSTD decode — zstd-sys-rs vs the zstd crate

Aggregate (all-core) decompression throughput on a real OSM corpus (WKB geometry bytes from the converted Europe extract, ~parquet-page-sized frames), decoding the same frames two ways: the stock zstd crate (one-shot, allocates per frame) and zstd-sys-rs's zero-copy path (a reused ZSTD_DCtx + reused output buffer). Both link the same static libzstd 1.5.7, so this validates the bindings and the zero-copy API on real data rather than chasing a codec difference. Needs OSM_GEOPARQUET (ZSTD_CHUNK_KB sets frame size).

(no bench results)

Quickstart

use std::collections::HashMap;
use std::sync::Arc;

use iceberg::io::LocalFsStorageFactory;
use iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation};
use skade_katalog::RedbCatalogBuilder;

# async fn run() -> anyhow::Result<()> {
let catalog = RedbCatalogBuilder::default()
    .db_path("/var/lib/nornir/catalog.redb")
    .warehouse_location("file:///var/lib/nornir/warehouse")
    .with_storage_factory(Arc::new(LocalFsStorageFactory))
    .load("nornir", HashMap::new())
    .await?;

let ns = NamespaceIdent::new("bench".to_string());
catalog.create_namespace(&ns, HashMap::new()).await?;

let schema = Schema::builder()
    .with_schema_id(0)
    .with_fields(vec![
        NestedField::required(1, "run_id", Type::Primitive(PrimitiveType::String)).into(),
        NestedField::required(2, "ops_sec", Type::Primitive(PrimitiveType::Double)).into(),
    ])
    .build()?;

let table = catalog
    .create_table(
        &ns,
        TableCreation::builder()
            .name("bench_runs".to_string())
            .schema(schema)
            .build(),
    )
    .await?;
# Ok(()) }

Multi-table atomic commits

Iceberg's per-table Transaction::commit only gives single-table atomicity. When a release logically spans several tables (e.g. publishing bench_runs, dep_graph, and components from a single CI run), readers can otherwise observe a half-published state.

redb gives every write transaction global atomicity across all of its tables. RedbCatalog::atomic_release(commits) exploits this: each TableCommit is staged (metadata blobs written to object storage), then a single redb write transaction performs an optimistic-concurrency check on every base pointer and either flips them all or none.

       prepare metadata files       single redb txn
TableCommit ──┐                    ┌─ check base = current ┐
TableCommit ──┼─► fileio.write_to ─┤  …for every table…    ├─► commit
TableCommit ──┘                    └─ swap pointer         ┘

If iceberg-rust later exposes TableCommit construction publicly, the ergonomic story improves. Until then, atomic_release is most useful when you build commits via crates that have direct access to internal helpers (e.g. a nornir release driver).

Storage layout

Inside the redb file:

Where <ns_path> is the dot-joined namespace ident ("a.b.c"). One redb file may host multiple logical catalogs by reusing different <catalog> prefixes; in practice you'll point each catalog at its own file.

Known shortcomings (0.1.3)

• Schema evolution is not callable yet. iceberg-rust 0.9.1 seals the TransactionAction trait (pub(crate)), so downstream code can't construct AddSchema / SetCurrentSchema. The on-disk metadata already supports schema evolution — only the writer API is sealed — so it works here unchanged once upstream exposes it (0.10). These Transaction actions do work today: update_table_properties, fast_append, replace_sort_order, update_location, update_statistics, upgrade_format_version.

• No orphan-file garbage collection. A commit that fails after writing metadata blobs to object storage but before the redb pointer-swap commits leaves those blobs behind — harmless but unreferenced. Reap them with Iceberg's standard orphan-file procedures; there is no built-in sweeper.

License

Licensed under Apache License, Version 2.0, matching the upstream Apache Iceberg project.