oxidelake-api 0.2.0

# OxideLake

[![CI](https://github.com/vyncint/oxidelake/actions/workflows/ci.yml/badge.svg)](https://github.com/vyncint/oxidelake/actions/workflows/ci.yml)
[![crates.io](https://img.shields.io/crates/v/oxidelake-runtime?label=crates.io)](https://crates.io/crates/oxidelake-runtime)
[![docs.rs](https://img.shields.io/docsrs/oxidelake-runtime)](https://docs.rs/oxidelake-runtime)
![license](https://img.shields.io/badge/license-Apache--2.0-blue)
![MSRV](https://img.shields.io/badge/MSRV-1.94.1-orange)

A GPU-accelerated, Arrow-native distributed analytical query engine, written in Rust. It reads and writes Parquet on object storage; a catalog, a metastore and transactions are out of scope, so it is a query engine over a lake rather than a lakehouse in the full sense.

**Status: internal phases 0–7 complete; published 0.2.0. Production readiness is milestone v0.2.0, and 0.2.0 did not finish it** — ten [Phase 8](docs/roadmap.md) lines remain open, three of them needing a CUDA device to verify on. What 0.2.0 does add is the ability to tell what a deployment is actually doing: a CPU-fallback counter, placement notes under `oxide explain`, query spans, a per-query log line and a Prometheus endpoint. The default build is pure CPU and the whole gate is green on Linux x86_64 and macOS arm64. Both GPU backends have executed on real hardware: **Metal** on Apple silicon (conformance-tested against stock DataFusion on an M4 Pro) and **CUDA** on an NVIDIA T4 since 2026-09-06, where every kernel was NVRTC-compiled and launched — and where a correctness bug in the aggregation kernel was found. [STATUS.md](STATUS.md) holds the verification matrix, component by component, and is the ledger ADR-0012 makes it. The specification the engine was built to is [docs/SPEC.md](docs/SPEC.md).

## Install

```bash
brew install vyncint/tap/oxidelake       # macOS and Linux: static binaries, no Rust toolchain
cargo install oxidelake-runtime          # or build from crates.io
```

Both give you the `oxide`, `oxide-scheduler` and `oxide-worker` binaries.
The Homebrew formula is generated by the release itself from the archives'
checksums, so it never describes a build that did not happen.

Or download a static archive for Linux (x86_64, aarch64, musl) or macOS
(Apple silicon, Intel) from the [latest release](https://github.com/vyncint/oxidelake/releases/latest),
verify it against the `.sha256` beside it, and put the three binaries on your
`PATH`. Using the engine as a library is `cargo add oxidelake-api`.

## What OxideLake is

OxideLake runs the same plans in two modes over the Apache Arrow columnar memory model:

- **Embedded mode** (DuckDB-like) — in-process DataFusion; operators exchange `Arc<RecordBatch>` over bounded async channels with no serialization.
- **Cluster mode** (Spark-like) — Apache DataFusion Ballista schedules stages across executors and streams shuffle partitions over Arrow Flight; OxideLake supplies the GPU operators, the placement rule and the plan codec that ships them to executors.

Acceleration backends: **CPU** (always available, the correctness reference), **NVIDIA CUDA** (opt-in feature, NVRTC-JIT at runtime), **Apple Metal** on unified memory (opt-in, macOS only, MSL compiled at runtime).

The differentiator is the operator layer: `GpuFilterExec` (fused filter + projection), `GpuHashJoinExec`, `GpuAggregateExec` and `GpuVectorDistanceExec` are DataFusion `ExecutionPlan`s with their own CUDA and Metal kernels behind one object-safe hardware abstraction. A placement rule rewrites eligible plan nodes for the target backend — `EXPLAIN` shows `GpuFilterExec[metal]` — and every operator still selects the *real* local backend at execution time and falls back per batch to the CPU reference, which is what keeps a heterogeneous cluster correct. Storage is Parquet pruned by DataFusion's statistics, page index and Bloom filters, with Arrow IPC for zero-decode spill and caches.

## Quickstart

Everything below runs as written (Rust 1.98 via `rust-toolchain.toml`; the workspace builds on 1.94.1 and up — or just `make quickstart`):

```bash
cargo build --release -p oxidelake-runtime          # or `cargo install oxidelake-runtime`

# 1M-row demo table: Parquet with Bloom filters on id and k, page statistics on
./target/release/oxide gen-data --rows 1000000 --out data/

# embedded SQL over it
./target/release/oxide sql -q "SELECT k, SUM(v) FROM t GROUP BY k ORDER BY k" --table t=data/

# vector search: l2_distance / cosine_distance are built-in SQL UDFs
./target/release/oxide sql -q "SELECT id, l2_distance(emb, [1.0,0.0,2.0,0.5,-1.0,0.25,3.0,0.0]) AS d \
  FROM t ORDER BY d LIMIT 5" --table t=data/

# what the plan looks like for a GPU cluster: placement tags without needing the GPU
./target/release/oxide explain -q "SELECT k, SUM(v) FROM t WHERE k >= 2 AND v < 4.0 GROUP BY k" \
  --table t=data/ --target cuda

# the terminal dashboard over a real query: plan DAG, per-operator telemetry, column profiles
./target/release/oxide tui -q "SELECT k, SUM(v) FROM t WHERE k >= 2 GROUP BY k" --table t=data/ --target cuda
```

### In-database inference (optional, `--features predict`)

Score a row against a trained model without leaving SQL. Off by default; it
pulls in a tensor library.

```bash
cargo build --release -p oxidelake-runtime --features predict

# an oxmera Sequential/Linear model saved with safetensors, whose header
# declares {"__metadata__": {"oxidelake.activation": "relu"}}
./target/release/oxide sql --table t=data/ \
  -q "SELECT id, predict('scorer.safetensors', emb) AS logits FROM t LIMIT 5"

# a checkpoint whose header says nothing: state the activation in the query
./target/release/oxide sql --table t=data/ \
  -q "SELECT id, predict('scorer.safetensors', emb, 'gelu') AS logits FROM t LIMIT 5"
```

```
+----+--------------------------+
| id | logits                   |
+----+--------------------------+
| 0  | [-31.006792, 3.9050333]  |
| 1  | [7.2003703, -4.237343]   |
+----+--------------------------+
```

It composes with everything else — including the vector search that was
already here, which is the point: OxideLake could find the nearest rows and
could not score them.

```bash
./target/release/oxide sql --table t=data/ -q \
  "SELECT id, l2_distance(emb, [1.0,0.0,2.0,0.5,-1.0,0.25,3.0,0.0]) AS dist,
          predict('scorer.safetensors', emb)[2] AS score
   FROM t ORDER BY dist LIMIT 5"
```

The model file is read for its architecture, not just its weights: `predict`
rebuilds an `oxmera::nn::Sequential` of `Linear` layers from the `0.weight`,
`1.weight`, … naming, and takes the activation between them from the
safetensors `__metadata__` key `oxidelake.activation` — `relu`, `gelu`,
`sigmoid`, `tanh` or `none`, applied between layers and never after the last.
A file that declares nothing is **refused**, because the alternative is
scoring it with an activation it was not trained with and returning a
confident wrong number; the third argument `predict(path, features, 'relu')`
is how you say it yourself, and it must agree with the file when the file has
an opinion. The CPU-only execution, and why this is a UDF rather than an
operator, are recorded in
[ADR-0015](docs/decisions/ADR-0015-in-database-inference-at-the-udf-layer.md).

`--target cpu|cuda|metal` picks the *placement* target; execution always uses the hardware that is present (operators planned for an absent GPU take the per-batch CPU path — exactly what cluster executors do with a scheduler's plan).

### Cluster mode

```bash
# OXIDE_CLUSTER_BACKEND declares the cluster's placement capability (default: cpu, no rewrites)
OXIDE_CLUSTER_BACKEND=cuda ./target/release/oxide-scheduler --port 50050 &
./target/release/oxide-worker --scheduler-port 50050 --port 50051 --grpc-port 50052 &
./target/release/oxide sql --cluster df://127.0.0.1:50050 \
  -q "SELECT k, SUM(v) FROM t GROUP BY k ORDER BY k" --table t=data/
```

The end-to-end test in `crates/oxidelake-runtime/tests/cli.rs` spawns exactly this topology on ephemeral ports and asserts the cluster output is byte-identical to embedded mode.

### DataFrame API

```rust
use oxidelake_api::prelude::*;

let session = OxideSession::local()?;
session.register_parquet("t", "data/").await?;
let nearest = session
    .table("t").await?
    .filter(col("k").gt_eq(lit(2)))?
    .vector_distance("emb", &[0.0; 8], DistanceMetric::L2, "d")?
    .sort(vec![col("d").sort(true, false)])?
    .limit(10)?
    .collect().await?;
```

The fluent verbs build ordinary DataFusion plans, so the placement rule lowers them the same way it lowers SQL — `.vector_distance(…)` plans `GpuVectorDistanceExec[metal]` on a Metal target.

### GPU backends

```bash
# CUDA: compiles and lints with no CUDA installed (cudarc dynamic loading + NVRTC)
cargo check -p oxidelake-runtime --features cuda

# Metal (macOS): the conformance suite executes on the device
cargo test -p oxidelake-compute --features metal -- --ignored

# On a CUDA machine (run on a T4 since 2026-09-06; see STATUS.md):
OXIDE_BACKEND=cuda cargo test -p oxidelake-compute --features cuda -- --ignored
```

## Configuration

Every knob OxideLake reads, in one table. Flags win over environment
variables; an explicit choice that the machine cannot honour is a startup
error, never a silent fallback.

### `oxide`

| Knob | Where | Default | What it does |
| --- | --- | --- | --- |
| `--query`, `-q` | `sql`, `explain`, `tui` | — | The SQL statement (optional for `tui`, which then shows the demo model). |
| `--table`, `-t` | `sql`, `explain`, `tui` | — | Registers `NAME=PATH` (a Parquet file or directory) before planning. Repeatable. |
| `--cluster` | `sql` | — | Runs on a Ballista scheduler (`df://host:port`) instead of in-process. |
| `--target` | `sql`, `explain`, `tui` | detected | Plans for `cpu`, `cuda` or `metal`. Placement only: execution uses the hardware that is present. Embedded mode only — on a cluster the scheduler decides. |
| `--batch-size` | `sql`, `explain`, `tui` | 8192, or 65536 on a GPU target | Rows per record batch. Batch boundaries never change results; larger batches amortise the host↔device round trip. `0` is refused. |
| `--output` | `sql` | `table` | `table`, `json` (an array of objects) or `csv` (RFC 4180 with a header). An empty result still prints the CSV header and `[]`, so a script can tell "no rows" from "the query failed". |
| `--rows` | `gen-data` | `1000000` | Rows in the demo table. |
| `--out` | `gen-data` | — | Output directory; writes `<out>/t.parquet`. |
| `--seed` | `gen-data` | `42` | PRNG seed. The same seed always writes the same bytes. |
| `--row-group-rows` | `gen-data` | `65536` | Rows per Parquet row group. |
| `--compression` | `gen-data` | `zstd` | `none`, `lz4` or `zstd`. |

### `oxide-scheduler`

| Knob | Default | What it does |
| --- | --- | --- |
| `--bind-host` | `127.0.0.1` | Address to bind. |
| `--port` | `50050` | gRPC port. |
| `--cluster-backend` | `$OXIDE_CLUSTER_BACKEND`, then `cpu` | The capability the cluster declares, which is what placement rewrites against. |
| `--metrics-port` / `--metrics-host` | off / `127.0.0.1` | As for the worker. A scheduler plans but does not execute, so its counters are empty; the endpoint exists so every process is scraped the same way. |

### `oxide-worker`

| Knob | Default | What it does |
| --- | --- | --- |
| `--scheduler-host` / `--scheduler-port` | `localhost` / `50050` | The scheduler to join. |
| `--port` | `50051` | Arrow Flight port for shuffle data. |
| `--grpc-port` | `50052` | gRPC control port. |
| `--concurrent-tasks` | available parallelism | Tasks run at once. |
| `--work-dir` | a temporary directory | Where shuffle files go. |
| `--metrics-port` / `--metrics-host` | off / `127.0.0.1` | Serve Prometheus metrics at `/metrics` (needs `--features metrics`). Unauthenticated: bind it on a private interface. Without the feature the flag is refused, never ignored. |
| `--backend` | `$OXIDE_BACKEND`, then detected | The backend this worker executes on. Selected before the first task, so it reaches the operators; a backend the machine cannot provide is a startup error. |

### Environment

| Variable | Read by | What it does |
| --- | --- | --- |
| `OXIDE_BACKEND` | every process that executes operators | Forces `cpu`, `cuda` or `metal` instead of the detected backend. `oxide-worker --backend` overrides it. |
| `OXIDE_CLUSTER_BACKEND` | `oxide-scheduler` | The cluster's declared placement capability (default `cpu`: no GPU rewrites). `--cluster-backend` overrides it. |
| `RUST_LOG` | every binary | `tracing` filter; logs go to stderr, so they never mix into `--output json`, and are uncoloured when stderr is not a terminal. `RUST_LOG=oxidelake_runtime=info` gives one line per query with the mode, rows, elapsed time and CPU fallbacks. |

## Repository map

| Path | Purpose |
|---|---|
| [`crates/oxidelake-core`](crates/oxidelake-core) | `EngineError`, `BackendKind`, operator parameter types, `TelemetryHub` |
| [`crates/oxidelake-memory`](crates/oxidelake-memory) | 64/128-byte-aligned buffers, pinned/UMA allocators, 3-tier `SpillManager` (a library: no query path registers batches with it in 0.x — see [architecture](docs/architecture.md#memory-model-and-spill-tiers)), Arrow IPC codec |
| [`crates/oxidelake-device`](crates/oxidelake-device) | object-safe `GpuBackend`; CPU reference, CUDA (NVRTC) and Metal (MSL) backends; `HardwareDetector` |
| [`crates/oxidelake-compute`](crates/oxidelake-compute) | the four `Gpu*Exec` operators, per-batch CPU fallback, `l2_distance`/`cosine_distance` UDFs, conformance suite |
| [`crates/oxidelake-storage`](crates/oxidelake-storage) | Parquet writer/pruning config with *proofs*, Arrow IPC spill files, io_uring `ObjectStore` (Linux — implemented and conformance-tested, not yet used by sessions), `gen-data` generator |
| [`crates/oxidelake-planner`](crates/oxidelake-planner) | `HardwarePlacementRule` (+ vector-distance lowering before projection pushdown), `OxidePhysicalCodec` |
| [`crates/oxidelake-runtime`](crates/oxidelake-runtime) | `OxideSession` (embedded + Ballista cluster), scheduler/worker wrappers, the `oxide` CLI, dashboard builder |
| [`crates/oxidelake-tui`](crates/oxidelake-tui) | ratatui dashboard, `TestBackend` snapshots and termlens PTY tests |
| [`crates/oxidelake-api`](crates/oxidelake-api) | `OxideFrame` fluent DataFrame API and prelude |
| [`crates/oxidelake-device/kernels/`](crates/oxidelake-device/kernels) | CUDA C (`.cu`, NVRTC at runtime) and MSL (`.metal`, `newLibraryWithSource`) sources — inside the device crate so the published crate is self-contained (`include_str!` cannot reach outside a crate's directory at publish time; ADR-0016) |
| [`docs/`](docs) | [roadmap](docs/roadmap.md) · [architecture](docs/architecture.md) · [dependencies](docs/dependencies.md) · [verification](docs/verification.md) · [ADRs](docs/decisions/README.md) |
| [`docs/SPEC.md`](docs/SPEC.md) | The executable spec the build agent executed, phase by phase |
| [`STATUS.md`](STATUS.md) | What is done, how it was verified, what was deferred — the honesty ledger |
| [`Makefile`](Makefile) · [`deny.toml`](deny.toml) | `make gate` and friends; supply-chain policy for `cargo deny` |

## Verification

Every change lands behind `required-green` and `commit-policy`. `make gate`
runs formatting, feature-specific Clippy, the test suites, all-feature rustdoc,
dependency-coherence and supply-chain checks, plus release and CI script tests.
[CI](.github/workflows/ci.yml) separates the Linux feature tests and checks the
MSRV; macOS runs Metal and PTY tests and executes conformance when a Metal
device exists. Dependency caching reuses compilation while tests still run;
release CI retains a clean full gate. Known documentation-only changes may
skip Rust jobs under the tested [verification policy](docs/verification.md).
The honesty rule applies throughout: nothing is claimed as working unless it
ran; GPU paths that only compiled are recorded as exactly that in
[STATUS.md](STATUS.md). The 2026-09-02 security and performance audit, with
every finding's status, is in [docs/audit-2026-09-02.md](docs/audit-2026-09-02.md).

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow (`make help` lists every target), [SECURITY.md](SECURITY.md) for the trust model and how to report vulnerabilities, and [CHANGELOG.md](CHANGELOG.md) for what changed.

## License

Apache-2.0 — see [LICENSE](LICENSE).