Tetration

For those who are more cur...

STILL IN DEVELOPMENT — layout v1 and query JSON/TOML may change before 1.0.

HDF5-shaped persistence (many large arrays in one durable file), Zarr-shaped chunking (regular grid, per-chunk compression, parallel I/O)—in a single mmap-friendly .tet file`, not a directory of shard blobs.

What it does today (v1)

• On-disk layout — superblock, dataset directory, chunk index, raw or zstd payloads (docs/layout_v1.md).
• Mmap + read planning — logical slices → chunk coordinates → ReadPlan.
• JSON / TOML query + execute — flat query documents (paired examples in fixtures/queries/), streaming reductions, tier-C stats, spill export, transform (zscore, minmax, l1/l2, center, scale, log1p, sqrt, softmax) and nan_mean / nan_std; named axes, coord label selection, QC counts (nan_count, null_count, inf_count, any_inf), covariance / correlation (docs/query_engine.md).
• Import / export — tet convert from HDF5, NetCDF, Zarr v3; tet export back to Zarr v3 (stored chunk bytes, nested groups).
• File health — tet verify (quick scan; --deep decodes every chunk), tet repair (plan / --apply safe fixes).
• CLI — tet info, tet verify, tet repair, tet query, tet qhist, tet convert, tet export.
• Optional GPU (Phase 10, experimental) — execution.device / tet query --device for tier-A/B f32 (and f16 on device); Metal (tetration-metal, macOS), CUDA (tetration-gpu), streaming fold + multi-GPU when host RAM does not fit a dense buffer. CPU streaming fold remains the default for large selections.
• C ABI — feature tetration-ffi, include/tetration.h, docs/ffi.md.

Wire dtypes (tags 1–10, row-major chunks): f32, f64, i32, i64, u8, u16, i16, u32, f16, u64. Booleans import as u8. See docs/layout_v1.md.

Quick start

macOS — Homebrew (recommended)

One-time tap (this repo ships Formula/tetration.rb; pulls in HDF5 and NetCDF for tet convert):

brew tap Latka-Industries/tetration https://github.com/Latka-Industries/tetration
brew install tetration
tet --help

Upgrade later: brew upgrade tetration.

From a local clone (no tap): brew install --build-from-source Formula/tetration.rb

cargo install

Default features need system HDF5 and NetCDF dev libraries (.h5 / .nc convert; Zarr v3 is Rust + bundled zstd):

cargo install tetration

Without HDF5/NetCDF libs: cargo install tetration --no-default-features — tet info / tet query on .tet files and Zarr import still work.

Build from source

git clone https://github.com/Latka-Industries/tetration.git
cd tetration
cargo build --release
export PATH="$PWD/target/release:$PATH"   # or: alias tet="$PWD/target/release/tet"

First commands

tet convert volume.h5 volume.tet          # HDF5 / NetCDF / Zarr v3 → .tet
tet export volume.tet volume.zarr/      # .tet → Zarr v3 directory (empty or new dir)

tet info volume.tet
tet verify volume.tet
tet verify --deep volume.tet -q    # full chunk decode (large files sample 128 by default)
tet query fixtures/queries/mean_temperature.toml -t volume.tet -x -q   # after convert; <name> from info
tet query '{"dataset":"<name>","inf_count":[]}' -t volume.tet -x -q

Daily driver: plan + execute with readable stdout:

tet query fixtures/queries/mean_temperature.toml -t data.tet -x -q
tet query q.json -t data.tet -x --format stats              # slim JSON (no chunk list)
tet query q.toml -t data.tet -x --format table --preview 6  # ASCII tables + slice grid
tet query q.json -t data.tet --format plan                  # catalog + read_plan only

Query documents are flat JSON or TOML (e.g. "mean": [] / mean = [], "spill": "slice.bin"); nested "operation" objects are rejected. Details: query document.

tet commands

docs/cli.md — tet info, tet verify, tet repair, tet query, tet qhist, tet convert, tet export (flags and subcommands). Live help: tet -h and tet <command> -h.

Documentation map

Design stance (short)

Partial I/O is the default case — mmap payload regions, touch only chunks that intersect the selection, parallel decode across disjoint tiles. Full-array loads into RAM are not required for planning or tier-A/B aggregates.

JSON/TOML is the control plane, not the storage encoding: hosts validate input, cap size, and enforce spill path policy (security notes).

Concurrency and scale

Read-many / write-once is the supported scale model for v1:

Not supported without extra coordination: multiple writers on one file, read-while-write, or two workers writing the same export spill path.

CPU workers: scale out with N processes × independent queries (or datasets), not by sharding one query inside the engine today. Phase 10 GPU uses the same chunk-parallel shape: dense materialize when RAM allows, else streaming device fold (gpu/streaming_fold.rs); cuda:multi / rocm:multi shard chunks across devices — see query engine — scalability.

Wire details: docs/layout_v1.md — Concurrency.

Non-goals (v1): SQL-on-files, arbitrary codec plugins, GPU codecs in the file format. Optional GPU: docs/query_engine.md.

Library use

[dependencies]
tetration = "0.1.7"

use tetration::prelude::*;
// TetWriterSession, TetFile, parse_query_json, parse_query_toml, execute_query_json, verify_tet_file, …

Embedder flow

1. Write — TetWriterSession::create → push_dataset → commit() (or commit_with_fill for streaming).
2. Read / aggregate — TetFile::open → execute_query_json → QueryResponse.

cargo run --example create_and_query
cargo run --example session_write

Query input: JSON or TOML

Flat JSON and TOML profiles compile to the same QueryDocument. tet query accepts .json / .toml paths, inline text, or stdin; leading { selects JSON, otherwise TOML (extension overrides).

{ "dataset": "temperature", "mean": [] }

dataset = "temperature"
mean = [] # scalar reduction

Library: parse_query_json, parse_query_toml, parse_query_text (auto-detect).

See Documentation map above for layout, query engine, FFI, fixtures, and docs.rs.

To do

• Python wrapper — separate repository (TBA); will pin crates.io tetration
• docs.rs examples — match on-disk guarantees when the format stabilizes