Tetration

For those who are more cur...

STILL IN DEVELOPMENT — layout v1 and query JSON may change before 1.0.

HDF5-shaped persistence (many large arrays in one durable file), Zarr-shaped chunking (regular grid, per-chunk compression, parallel I/O)—in a single mmap-friendly .tet file`, not a directory of shard blobs.

What it does today (v1)

• On-disk layout — superblock, dataset directory, chunk index, raw or zstd payloads (docs/layout_v1.md).
• Mmap + read planning — logical slices → chunk coordinates → ReadPlan.
• JSON query + execute — flat query documents, streaming reductions, tier-C stats, spill export (docs/query_engine.md).
• Import — tet convert from HDF5, NetCDF, Zarr v3 directory stores.
• CLI — tet info, tet query, tet qhist, tet convert.

Dtypes on disk and in query execution: f32, f64, i32, i64.

Quick start

Requires Rust 1.95+ for cargo install; the tet binary installs to ~/.cargo/bin (or Homebrew’s prefix when using brew).

macOS — Homebrew (recommended)

One-time tap (this repo ships Formula/tetration.rb; pulls in HDF5 and NetCDF for tet convert):

brew tap thicclatka/tetration https://github.com/thicclatka/tetration
brew install tetration
tet --help

Upgrade later: brew upgrade tetration.

From a local clone (no tap): brew install --build-from-source Formula/tetration.rb

cargo install

Default features need system HDF5 and NetCDF dev libraries (.h5 / .nc convert; Zarr v3 is Rust + bundled zstd):

cargo install tetration

Without HDF5/NetCDF libs: cargo install tetration --no-default-features — tet info / tet query on .tet files and Zarr import still work.

Build from source

git clone https://github.com/thicclatka/tetration.git
cd tetration
cargo build --release
export PATH="$PWD/target/release:$PATH"   # or: alias tet="$PWD/target/release/tet"

First commands

tet convert volume.h5 volume.tet          # HDF5 / NetCDF / Zarr v3 → .tet

tet info volume.tet
tet query '{"dataset":"<name>","mean":[]}' -t volume.tet -x -q   # <name> from info output

Daily driver: plan + execute with readable stdout:

tet query q.json -t data.tet -x -q              # one-line aggregate
tet query q.json -t data.tet -x --format stats  # slim JSON (no chunk list)
tet query q.json -t data.tet --format plan      # catalog + read_plan only

Query JSON is flat (e.g. "mean": [], "spill": "slice.bin"); nested "operation" objects are rejected. Details: query document.

tet commands

Full flag lists: tet -h and tet <command> -h (always match the installed binary).

tet info

tet query

QUERY: path to .json, inline JSON, - for stdin, or omit to read stdin.

tet qhist

Stored under the platform cache (query_history.jsonl), not in the .tet file. Env: TET_NO_QUERY_HISTORY, TET_QUERY_HISTORY_FILE, TET_QUERY_HISTORY_MAX. Details: GETTING_STARTED.md — qhist.

tet convert

More examples and roadmap: GETTING_STARTED.md.

Documentation map

Design stance (short)

Partial I/O is the default case — mmap payload regions, touch only chunks that intersect the selection, parallel decode across disjoint tiles. Full-array loads into RAM are not required for planning or tier-A/B aggregates.

JSON is the control plane, not the storage encoding: hosts validate input, cap size, and enforce spill path policy (security notes).

Non-goals (v1): SQL-on-files, arbitrary codec plugins, GPU codecs in the file format. GPU use is “materialize on CPU (or spill), then copy to device” in bindings—see Phase 9 in GETTING_STARTED.md. Python wheels and a narrow C ABI are planned (Phase 10); the layout spec is the portable floor.

Library use

[dependencies]
tetration = "0.1"

use tetration::prelude::*;
// or: tetration::layout::mmap_file_read, tetration::query::{parse_query_json, …}

Embedders get the full QueryResponse; the CLI uses format_query_response for stdout modes. Today: low-level writers (tetration::catalog) and query plan/execute (tetration::prelude + tetration::query); Phase 7 adds documented create + in-process execute workflows — see GETTING_STARTED.md — Phase 7.