Tetration

For those who are more cur...

STILL IN DEVELOPMENT — layout v1 and query JSON may change before 1.0.

HDF5-shaped persistence (many large arrays in one durable file), Zarr-shaped chunking (regular grid, per-chunk compression, parallel I/O)—in a single mmap-friendly .tet file`, not a directory of shard blobs.

What it does today (v1)

• On-disk layout — superblock, dataset directory, chunk index, raw or zstd payloads (docs/layout_v1.md).
• Mmap + read planning — logical slices → chunk coordinates → ReadPlan.
• JSON query + execute — flat query documents, streaming reductions, tier-C stats, spill export; named axes, coord label selection, QC counts (nan_count, null_count, inf_count), covariance / correlation (docs/query_engine.md).
• Import / export — tet convert from HDF5, NetCDF, Zarr v3; tet export back to Zarr v3 (stored chunk bytes, nested groups).
• File health — tet verify (quick scan; --deep decodes every chunk), tet repair (plan / --apply safe fixes).
• CLI — tet info, tet verify, tet repair, tet query, tet qhist, tet convert, tet export.

Wire dtypes (tags 1–10, row-major chunks): f32, f64, i32, i64, u8, u16, i16, u32, f16, u64. Booleans import as u8. See docs/layout_v1.md.

Quick start

macOS — Homebrew (recommended)

One-time tap (this repo ships Formula/tetration.rb; pulls in HDF5 and NetCDF for tet convert):

brew tap thicclatka/tetration https://github.com/thicclatka/tetration
brew install tetration
tet --help

Upgrade later: brew upgrade tetration.

From a local clone (no tap): brew install --build-from-source Formula/tetration.rb

cargo install

Default features need system HDF5 and NetCDF dev libraries (.h5 / .nc convert; Zarr v3 is Rust + bundled zstd):

cargo install tetration

Without HDF5/NetCDF libs: cargo install tetration --no-default-features — tet info / tet query on .tet files and Zarr import still work.

Build from source

git clone https://github.com/thicclatka/tetration.git
cd tetration
cargo build --release
export PATH="$PWD/target/release:$PATH"   # or: alias tet="$PWD/target/release/tet"

First commands

tet convert volume.h5 volume.tet          # HDF5 / NetCDF / Zarr v3 → .tet
tet export volume.tet volume.zarr/      # .tet → Zarr v3 directory (empty or new dir)

tet info volume.tet
tet verify volume.tet
tet verify --deep volume.tet -q    # full chunk decode (large files sample 128 by default)
tet query '{"dataset":"<name>","mean":[]}' -t volume.tet -x -q   # <name> from info output
tet query '{"dataset":"<name>","inf_count":[]}' -t volume.tet -x -q

Daily driver: plan + execute with readable stdout:

tet query q.json -t data.tet -x -q              # one-line aggregate
tet query q.json -t data.tet -x --format stats  # slim JSON (no chunk list)
tet query q.json -t data.tet --format plan      # catalog + read_plan only

Query JSON is flat (e.g. "mean": [], "spill": "slice.bin"); nested "operation" objects are rejected. Details: query document.

tet commands

Full flag lists: tet -h and tet <command> -h (always match the installed binary).

tet info

tet verify

Exit code 1 when verification fails (CI-friendly). Manual smoke fixtures: fixtures/small/tet/README.md.

tet repair

tet query

QUERY: path to .json, inline JSON, - for stdin, or omit to read stdin.

tet qhist

Stored under the platform cache (query_history.jsonl), not in the .tet file. Env: TET_NO_QUERY_HISTORY, TET_QUERY_HISTORY_FILE, TET_QUERY_HISTORY_MAX. Details: GETTING_STARTED.md — qhist.

tet convert

tet export

Preserves per-dataset raw or zstd chunk bytes; slash-separated dataset names become nested groups (primary/f32). Library: export_tet_to_zarr.

More examples and roadmap: GETTING_STARTED.md.

Documentation map

Design stance (short)

Partial I/O is the default case — mmap payload regions, touch only chunks that intersect the selection, parallel decode across disjoint tiles. Full-array loads into RAM are not required for planning or tier-A/B aggregates.

JSON is the control plane, not the storage encoding: hosts validate input, cap size, and enforce spill path policy (security notes).

Non-goals (v1): SQL-on-files, arbitrary codec plugins, GPU codecs in the file format. GPU use is “materialize on CPU (or spill), then copy to device” in bindings—see Phase 10 in GETTING_STARTED.md. Phases 8–9 are done (verify/repair, wire dtypes through u64/f16, named axes, coord labels, histogram edges, QC counts, covariance/correlation, tet export). Next: Phase 10 GPU hooks, Phase 11 Python wheels + narrow C ABI; the layout spec is the portable floor.

Library use

[dependencies]
tetration = "0.1"

use tetration::prelude::*;
// TetWriterSession, TetFile, parse_query_json, execute_query_json, verify_tet_file, …

Rust API by phase

Phases 0–9 below are done in this repo unless marked later. Full checklist: GETTING_STARTED.md.

Typical embedder flow (Phase 7 on top of Phase 1 + 4):

1. Write — TetWriterSession::create → push_dataset → commit() (or commit_with_fill for streaming tiles).
2. Read / aggregate — TetFile::open → execute_query_json → QueryResponse (CLI: format_query_response for stdout).

Examples: cargo run --example create_and_query, session_write, inspect_catalog (see src/catalog/session.rs). Smoke fixtures: cargo run --example gen_small_tet_fixtures → fixtures/small/tet/.