# Tetration on-disk layout — version 1
This document describes the **v1** `.tet` container: a fixed **32-byte superblock** at offset 0, followed (when `dataset_count > 0`) by a **dataset directory blob**, an **8-byte-aligned** **chunk index** region, and **raw chunk payloads**.
## File map
Byte offsets increase left → right. All integers are **little-endian**.
`align8(n) = (n + 7) & !7` (round up to an 8-byte boundary).
### Regions at a glance
| # | Region | Starts at | Size | When present |
| --- | --------------------- | -------------------------------------------------------- | ------------------------------------------------- | -------------------------------------------------------- |
| 1 | **Superblock** | `0` | `32` B fixed | always |
| 2 | **Dataset directory** | `32` | `8 + dataset_blob_len` | `dataset_count > 0` |
| 3 | **Padding** | `40 + dataset_blob_len` | `0 … 7` B | only bytes needed so the next region is 8-byte aligned |
| 4 | **Chunk index** | `chunk_index_offset` (= `align8(40 + dataset_blob_len)`) | `chunk_index_length` (= `32 + entry_count × 104`) | `dataset_count > 0` |
| 5 | **Chunk payloads** | each `payload_offset` from index | each `stored_byte_len` | one span per index row; may be non-contiguous in general |
**Empty file** (`dataset_count = 0`): regions 2–5 are absent; `chunk_index_offset = 32`, `chunk_index_length = 0`; the file may end at byte 32.
**Populated file**: the reference writer packs payloads after the index, but v1 only requires `payload_offset + stored_byte_len ≤ file_len` for every row.
**How pieces connect:** the superblock’s `chunk_index_offset` / `chunk_index_length` bound region ④. Each index row’s `payload_offset` points at one payload span in region ⑤. Dataset metadata in region ② supplies `shape`, `chunk_shape`, and `dtype` for interpreting those bytes (see [query engine](query_engine.md) for read planning).
## Endianness and alignment
- All multi-byte integers are **little-endian** (`u32`, `u64`).
- The superblock is **32 bytes**. Chunk index entries and payload offsets assume **8-byte alignment** for index base offsets computed from the dataset directory.
## Magic and `layout_version`
- Bytes `0..4` must be ASCII **`TETR`**.
- Bytes `4..8` are **`layout_version`** (`u32` LE). Only **`1`** is defined today.
This matches the optional `layout_version` field in JSON/TOML query documents (`tetration::query::QueryDocument`).
## Superblock v1 (32 bytes)
| Offset | Size | Type | Field | Notes |
| ------ | ---- | -------- | -------------------- | ------------------------------------------------------------------------------- |
| 0 | 4 | `[u8;4]` | `magic` | Must be `TETR`. |
| 4 | 4 | `u32` | `layout_version` | Must be `1`. |
| 8 | 4 | `u32` | `dataset_count` | Number of dataset records in the directory blob; **0** means no directory blob. |
| 12 | 4 | `u32` | `flags` | Bit **`1`**: optional **history footer** at EOF (see below). Otherwise **0**. |
| 16 | 8 | `u64` | `chunk_index_offset` | Byte offset to the **chunk index** region (see below). |
| 24 | 8 | `u64` | `chunk_index_length` | Length in bytes of the chunk index region. |
### Empty file (`dataset_count = 0`)
- `chunk_index_offset = 32`
- `chunk_index_length = 0`
- The file may end at byte 32 (no trailing bytes required).
Readers must ensure `chunk_index_offset + chunk_index_length` fits in the file and does not overflow.
**Typical populated file** (contrast with empty above): superblock with `dataset_count = N`, dataset records at offset 40, chunk index at `align8(40 + dataset_blob_len)`, payloads referenced by index rows.
## Dataset directory (only when `dataset_count > 0`)
When `dataset_count > 0`, bytes starting at offset **32** are:
| Offset | Size | Field | Notes |
| ------ | ---- | ------------------ | --------------------------------------------------------------------------------------- |
| 32 | 8 | `dataset_blob_len` | Length in bytes of the concatenated **dataset records** that immediately follow. |
| 40 | \* | `dataset_records` | Total size = `dataset_blob_len`. Must contain exactly `dataset_count` records in order. |
### Dataset record (concatenated, variable length per record)
Each record is:
| Field | Type | Notes |
| ------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `name_len` | `u32` LE | Byte length of UTF-8 `name` that follows. |
| `dtype` | `u32` LE | `1` = **`f32`**, `2` = **`f64`**, `3` = **`i32`**, `4` = **`i64`**, `5` = **`u8`**, `6` = **`u16`**, `7` = **`i16`**, `8` = **`u32`**, `9` = **`f16`**, `10` = **`u64`**, row-major within each chunk. |
| `ndim` | `u32` LE | Rank in `1 … 8`. |
| `reserved` | `u32` LE | Write **0**. |
| `name` | `[u8]` | UTF-8 of length `name_len`. |
| _padding_ | `0…7` | Zero bytes so the **total byte length from the start of this record** to the first byte of `shape[0]` is a multiple of **8**. |
| `shape` | `ndim × u64` LE | Global array shape. |
| `chunk_shape` | `ndim × u64` LE | Chunk size along each axis (must tile `shape`; `write_one_chunk_raw_file` requires a single tile `chunk_shape == shape`; `write_raw_array_file` supports regular multi-chunk grids). |
Records are concatenated in catalog order; `dataset_id` in the chunk index is the **0-based index** into this list (`0` = first record).
**Booleans** are not a separate wire tag: import maps HDF5 `Boolean`, Zarr `bool`, and similar to **`u8`** (0/1). Layout v2 is reserved for changes that cannot extend v1.
### Axis metadata (Phase 7 baseline)
v1 dataset records carry **`shape`** and **`chunk_shape`** only — no axis names or per-index labels on disk yet. Planned metadata (file header blob, dataset attrs, or sidecar regions) distinguishes two **separate** layers:
| Layer | What it names | Count | Example (3D weather) | Example (2D table) |
| --------------------- | ------------------------------------------- | --------------------- | ------------------------------------ | --------------------------------- |
| **Dimension names** | Each **axis** (the “major” row/column role) | **`ndim` strings** | `time`, `lat`, `lon` | `row`, `column` |
| **Coordinate labels** | Each **position** along one axis | **`shape[d]` values** | timestamps, latitudes, station codes | row 0 = `Alice`, col 2 = `salary` |
Do **not** conflate them:
- **Dimension name** — “axis 0 is called **time**” → enables `"mean": "time"` (JSON/TOML) instead of `"mean": 0`.
- **Coordinate label** — “index 42 along **time** is **2024-03-15**” → enables slice/filter by value, alignment across datasets, and (with extra ops) group-by keys.
**Analogues:** NetCDF dimension name vs coordinate variable; pandas `Index.name` vs `Index` values; xarray `dims` vs `coords`.
**Planned storage sketch** (informative, not wire-final):
```json
{
"dim_names": ["time", "station"],
"coords": {
"time": {
"dtype": "i64",
"storage": "inline | dataset_ref | payload_offset"
},
"station": { "dtype": "string", "storage": "…" }
},
"attrs": { "units": "K", "long_name": "surface temperature" }
}
```
- **`dim_names`** — small, always safe in header/catalog attrs.
- **`coords`** — may be **O(n)** along an axis; large coordinate vectors may live as a **1D dataset** in the same file, a metadata chunk, or inline when small.
- **`attrs`** — CF-style scalar metadata (units, `long_name`); not an index.
**Coordinate labels ≠ a query index by default.** Storing labels enables **name → integer index** resolution at plan time. Fast **filter / group-by** on high-cardinality keys may additionally need an auxiliary lookup structure (sorted coords, hash map, optional on-disk index) — a separate layout/query decision, not automatic from storing strings.
See [query engine — dimension names vs coordinates](query_engine.md#dimension-names-vs-coordinate-labels).
## Chunk index region
- Starts at `chunk_index_offset`, which **must** equal `align8(40 + dataset_blob_len)` when `dataset_count > 0`.
- `align8(n) = (n + 7) & !7`.
### Chunk index header (32 bytes)
| Offset | Size | Field | Notes |
| ------ | ---- | --------------------------- | ------------------------------------------------------------------------- |
| 0 | 4 | magic | ASCII **`TIDX`**. |
| 4 | 4 | `index_version` | `u32` LE; must be **1**. |
| 8 | 8 | `entry_count` | Number of fixed-size entries. |
| 16 | 2 | `memory_budget_percent_bps` | `u16` LE; basis points (10000 = 100%). **0** = engine default (25%). |
| 18 | 2 | reserved | Write **0**. |
| 20 | 4 | `memory_budget_bytes` | `u32` LE fixed RAM cap for dense decode; **0** = use percent of host RAM. |
| 24 | 8 | reserved | Write **0**. |
These fields map to [`FileExecutionSettingsV1`](../src/catalog/execution.rs). Writers set them via **`RawArrayWrite::file_execution`**; readers surface them on **`catalog.file_execution`** in query responses. Query JSON **`execution.memory_budget_*`** overrides file defaults at runtime (see [query engine — memory budget](query_engine.md#memory-budget-and-execution-strategies)). Basis points: **10000 = 100%** (engine default when both are zero: **2500 = 25%** of detected host RAM).
The total chunk index byte length must be exactly:
`32 + entry_count * entry_wire_len`
### Chunk index entry (fixed `entry_wire_len` bytes, LE)
Layout (see `ChunkIndexEntryV1::WIRE_LEN` in `src/catalog/index.rs`):
- `dataset_id`: `u64`
- `chunk_index[8]`: eight `u64` values — chunk grid coordinates `i0 … i7`; unused slots are **0** (rank comes from the dataset record’s `ndim`).
- `payload_offset`: `u64` — file offset to **stored** bytes for this chunk.
- `raw_byte_len`: `u64` — logical uncompressed size of the chunk tensor in bytes.
- `stored_byte_len`: `u64` — bytes on disk at `payload_offset`.
- `codec`: `u32` — **`0`** = raw copy (`stored_byte_len` **must** equal `raw_byte_len`). **`1`** = **zstd**–compressed payload at `payload_offset`; `stored_byte_len` is the compressed size on disk and **`raw_byte_len`** is the uncompressed tensor size in bytes after decode.
- `reserved`: `u32` — **0**.
Each entry is **104 bytes** on the wire (`ChunkIndexEntryV1::WIRE_LEN`).
Payload bytes must lie fully inside the file: `payload_offset + stored_byte_len ≤ file_len`.
**Index on disk:** `TIDX` header (32 B) then `entry_count` fixed-size rows (104 B each). See field list above.
**Reader resolution** (how a query reaches bytes; details in [query engine](query_engine.md)):
```mermaid
flowchart LR
Q["dataset name"] --> D["dataset_id"]
D --> G["chunk coords"]
G --> L["index row"]
L --> R["mmap payload"]
```
Each index row’s `payload_offset` selects the byte span in region ⑤.
### Contiguous raw payloads (query fold hint)
For a **full dense** selection over a dataset, the query engine may treat all touched **raw** (`codec = 0`) payloads as one logical byte stream when:
1. Each chunk’s `payload_offset + raw_byte_len` equals the next chunk’s `payload_offset` (in read-plan order).
2. The sum of `raw_byte_len` equals the logical selection size in bytes.
When that holds and the working set is **out-of-core** relative to host RAM, tier-A/B scalar folds can use **linear scan** — sequential 64 MiB windows over the hyperslab — instead of parallel per-chunk mmap. See [query engine — adaptive fold I/O](query_engine.md#adaptive-fold-io). **Zstd** chunks (`codec = 1`) or non-contiguous payload layout still use per-chunk decode/fold.
Reference writers (`write_raw_array_file`) append payloads sequentially, so converted multi-chunk grids typically satisfy (1)–(2) for full-file scans.
### Per-chunk payload codecs
| Codec | `stored_byte_len` vs `raw_byte_len` | On-disk bytes |
| ---------- | ------------------------------------------- | -------------------------------------------------------------- |
| **0** raw | must be equal | tensor payload (LE **`f32`** or **`f64`** per dataset `dtype`) |
| **1** zstd | `stored` = compressed, `raw` = decoded size | zstd frame; decode to `raw_byte_len` bytes |
See also [`query_engine.md`](query_engine.md) for how the query engine materializes planned chunks.
## Optional history footer (v1 extension)
When superblock **`flags & 1`**, the file may end with a self-describing footer **after** all chunk payloads:
| Region (suffix at EOF) | Size | Notes |
| ---------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `history_json` | variable | UTF-8 JSON object: `{"history":[…], "metadata":{…}}` (optional keys). See below. |
| `metadata_spill` | variable | Optional raw UTF-8 JSON for `metadata` when inline `history_json` would exceed **64 KiB**; referenced by `metadata_ref` in `history_json`. Lies after chunk payloads, before `history_json`. |
| `history_json_len` | 8 | `u64` LE byte length of `history_json`. |
| `history_version` | 4 | `u32` LE; must be **1**. |
| magic | 4 | ASCII **`THST`**. |
Chunk payload validation uses **`file_len − footer_suffix`** (footer JSON + tail + optional `metadata_spill`). Readers without history support ignore the flag and treat the full file length as payload bounds only when the flag is **0**. `tet info` / [`read_tet_summary_v1`](../src/catalog/mod.rs) surface parsed `history` and resolved `metadata`.
**`history` rows** — each entry is a JSON object `{ "op", "source", "at", "parents"?, "params"? }`. Legacy files may use a three-element array `[op, source, at]`; readers accept both. New writers emit objects only.
**`metadata`** — inline object when small (see [`MetadataLimitsV1`](../src/catalog/metadata.rs)). When too large, writers spill UTF-8 JSON to `metadata_spill` and set `"metadata_ref": { "offset", "len" }` in `history_json` (mutually exclusive with inline `metadata`).
## Reference subset (current Rust writer)
The `write_one_chunk_raw_file` helper in `tetration::catalog` writes exactly **one** dataset and **one** chunk: `chunk_shape` must equal `shape` so the chunk grid has a single tile; payloads are always **raw** (`codec = 0`). **`dtype`** may be any supported element tag (`1`–`10`).
`write_raw_array_file` / `RawArrayWrite` accept per-chunk **`chunk_codec`**: compare to **`CHUNK_PAYLOAD_CODEC_V1.raw`** (**0**, raw tiles) or **`CHUNK_PAYLOAD_CODEC_V1.zstd`** (**1**, zstd-compressed frames; `stored_byte_len` may differ from `raw_byte_len`). **`dtype`** may be any supported element tag (`1`–`10`). Optional **`file_execution`** writes TIDX execution settings (memory budget). The Rust API exposes this as the `ChunkPayloadCodecV1` struct plus the `CHUNK_PAYLOAD_CODEC_V1` constant in `tetration::catalog`; decode symmetry via **`ChunkPayloadCodecV1::decode_tile_payload`**.
## Concurrency (informative)
v1 has **no** on-disk locking, lease, or live-append protocol. Scale comes from **immutable files + mmap readers**, not from concurrent writers.
| Pattern | Supported? | Notes |
| ------------------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Sealed file, many readers** | Yes (intended) | After `commit` / `tet convert` completes, any number of hosts may mmap read-only and query. Chunk index + payload offsets are fixed. |
| **One writer, then readers** | Yes | Reference writers (`write_raw_array_file`, [`TetWriterSession`](../src/catalog/session.rs)) produce a stable byte image. |
| **Parallel writers on one file** | No (undefined) | Exclusive create or a single coordinated writer; parallel payload append without a spec can corrupt the index. |
| **Read while another process writes** | No (undefined) | Readers may see torn index rows or short payloads. |
| **Many queries, one file** | Yes | Independent plans; per-query temp spills are unique; user export `spill` paths must not collide. |
**Inside one query**, the engine already parallelizes chunk I/O on CPU when in-core (Rayon) or uses a sequential linear scan when out-of-core ([query engine — adaptive fold I/O](query_engine.md#adaptive-fold-io)).
Full deployment notes: [README — Concurrency and scale](../README.md#concurrency-and-scale).
## File health (verify / repair)
**`tet verify`** (library: [`verify_tet_file`](../src/verify/mod.rs)) checks superblock, dataset directory, chunk index, payload bounds, optional decode integrity (quick scan: first **128** chunks on large files; **`--deep`** / `VerifyOptions::deep_decode` for all chunks), dataset tensor-byte cross-check, and footer/metadata limits.
**`tet repair`** ([`repair_tet_file`](../src/repair/mod.rs)) applies safe in-place fixes from verify recommendations (today: **`footer_invalid`** — truncate after chunk payloads when the `THST` JSON is corrupt).
Tracked manual smoke files: [`fixtures/small/tet/README.md`](../fixtures/small/tet/README.md). Regenerate: `mise run fixtures:small-tet` or `cargo run --example gen_small_tet_fixtures`.