# Query engine
The **query engine** (`src/query/`) turns a validated [`QueryDocument`](../src/query/types/document.rs) into a [`QueryResponse`](../src/query/types/response.rs): catalog resolution, a chunk-level **read plan**, and optionally mmap-backed decode plus **`operation`** aggregates.
**Input:** flat **JSON** or **TOML** profiles (`parse_query_json`, [`parse_query_toml`](../src/query/document_toml.rs), [`parse_query_text`](../src/query/document.rs) with auto-detect). TOML is converted to a JSON value tree, then the same wire parser runs. The canonical wire shape is defined in **`document_wire.rs`**; size/depth limits and **`validate_query`** live in **`document.rs`**. Wire types live in **`types/`** (`document`, `plan`, `response`, `error`). Planning and decode live in **`plan/`** and **`decode/`**; materialize, fold, and dtype routing in **`materialize/`**, **`fold/`**, and **`dispatch.rs`**; orchestration in **`engine/`** (`run`, `operations`, `budget`, `spill_policy`). The engine assumes a parsed, validated document and a mmap’d `.tet` byte slice when planning against a file.
See [JSON security (input and output)](#json-security-input-and-output) for trust boundaries and hardening notes (apply to TOML input as well).
## End-to-end flow
```mermaid
flowchart TD
subgraph control["Control plane outside engine"]
In["Query JSON or TOML"]
Parse["parse_query_text / parse_query_json / parse_query_toml"]
Validate["validate_query"]
In --> Parse --> Validate
end
subgraph entry["engine run.rs"]
PQ["plan_query_empty"]
PQM["plan_query_with_tet_mmap"]
Validate --> PQ
Validate --> PQM
end
subgraph catalog_mmap["catalog mmap"]
Summary["read_tet_summary_v1"]
Match{"dataset in catalog?"}
Chunks["chunk_coords_intersecting_strided"]
end
subgraph planning["plan/"]
Sel["selection resolved_dense_global_box"]
RP["read_plan build_read_plan"]
PQM --> Summary --> Match
Match -->|no| Unmatched["DatasetResolution unmatched"]
Match -->|yes| Sel --> Chunks --> RP
end
subgraph exec["Optional execution"]
Gate{"preview limit set?"}
BEP["build_execution_preview"]
RP --> Gate
Gate -->|no| RespPlan["QueryResponse read_plan only"]
Gate -->|yes| BEP
end
subgraph out["Response"]
RespFull["QueryResponse full"]
Unmatched --> RespFull
RespPlan --> RespFull
BEP --> RespFull
PQ --> Echo["QueryResponse plan-only echo"]
end
```
**CLI mapping:** `tet query [QUERY]` reads **`.json` / `.toml`**, inline JSON/TOML, or stdin (`-` or omit). Format: file extension **`.json` / `.toml`**, else leading **`{`** → JSON, otherwise TOML ([`detect_query_input_format`](../src/query/document.rs)). Then `validate_query` and `plan_query_empty` or `plan_query_with_tet_mmap_ex`. **`-t PATH`** supplies the mmap; **`-x` / `--execute`** runs `build_execution_preview` and sets `raw_f32_preview_max` from **`--preview`** (alias **`--preview-f32`**): default **64** for **`--format full|json`**, **0** for **`stats|quiet|table`** when omitted; explicit **`0`** with an **`operation`** skips preview arrays but still aggregates; explicit **`0`** with top-level **`"spill"`** (no reduction key) still runs **`mmap_spill`** export. **`tet qhist`** still stores query JSON in the platform cache, not TOML.
Success stdout is formatted by [`format_query_response`](../src/query/cli/output/mod.rs) (presentation only — the in-memory [`QueryResponse`](../src/query/types/response.rs) is unchanged for embedders).
| `--format` | Short | stdout |
| ---------------- | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `full` (default) | — | Pretty JSON, full `QueryResponse` |
| `json` | — | Compact one-line JSON, full envelope |
| `stats` | — | Slim JSON: status, catalog/read_plan summary, `execution` aggregates — no `read_plan.chunks[]`, no preview arrays |
| `plan` | — | Slim JSON: catalog + `read_plan` summary only (no `chunks[]`, no `execution`; **`-x`** still runs decode — use without **`-x`** for plan-only) |
| `quiet` | `-q` | One line: `dataset=… status=… op=…` + primary aggregate (best after **`-x`** with `operation`) |
| `table` | — | ASCII tables: query summary, `read_plan`, execution I/O, scalar/partial **result**, optional **preview** sample (like `tet info`) |
**Phase 10 (device routing):** optional `execution.device` or `tet query … --device` — see [Phase 10 — optional GPU](#phase-10--optional-gpu-experimental).
Errors always go to **stderr** with non-zero exit. See [CLI query output](#cli-query-output).
## Phase 10 — optional GPU (experimental)
**Status:** merged on `main` ([PR #12](https://github.com/Latka-Industries/tetration/pull/12)). Routing, CUDA/Metal/ROCm kernels, **streaming per-chunk GPU fold**, decode∥GPU pipeline, and multi-GPU sharding ship behind optional crate features. **CPU streaming fold remains the fast, correct default** for large and extra-large selections on typical Apple Silicon benches.
### What it does today
1. **Host decode** — mmap + chunk decode unchanged (raw **0** / zstd **1**).
2. **GPU path (when routed)** — tier-A/B **`f32`** or **`f16`** (promoted to **`f32`** per chunk on device):
- **Dense:** host **`f32`** materialize when the host-RAM gate allows, then block-reduce on device for **`sum`**, **`mean`**, **`min`**, **`max`**, **`count`** (`tetration-metal` / `tetration-gpu` / `tetration-rocm`).
- **Streaming:** per-chunk decode + device partials + host merge ([`streaming_fold.rs`](../src/query/gpu/streaming_fold.rs)) when a full dense buffer does not fit; optional decode∥GPU pipeline when `chunk_count > 1`.
- **Multi-GPU:** `cuda:multi` / `rocm:multi` shard chunks across visible devices ([`multi.rs`](../src/query/gpu/multi.rs)).
- **Host (`f64` SIMD):** population **`var`** / **`std`** (`ddof = 0`) — per-chunk on the streaming path; dense path uses host SIMD after materialize (GPU f32 sumsq was numerically wrong at ~10⁹ elements).
3. **CPU streaming** — parallel chunk fold or out-of-core linear scan when GPU is not selected or falls back.
Preview JSON / **`stats`**: `device_requested`, `device_used`, `device_fallback_reason`, `device_gpu_reduce`, `device_gpu_pipeline`, `device_gpu_multi`.
### `execution.device` / `--device`
| Token | Behavior |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cpu` | Always host streaming fold. |
| `auto` | Metal on macOS when `tetration-metal` enabled, else CUDA device 0 when `tetration-gpu` enabled, else CPU. Skips GPU below **64 MiB** logical selection. |
| `metal` | Apple GPU when feature enabled; else CPU + `gpu_feature_disabled`. |
| `cuda` / `cuda:N` | NVIDIA GPU when feature enabled; else CPU + `gpu_feature_disabled`. |
CLI **`--device`** overrides query JSON `execution.device` (needs **`-x`**).
### Host-RAM gate (before materialize)
GPU scalar paths need a dense host buffer of size `logical_f32_bytes`. Routing refuses device when:
- logical bytes **>** **85%** of probed host RAM (`MemAvailable` / macOS free+inactive), or
- logical bytes **>** **8 GiB** when RAM cannot be probed.
When the selection exceeds the host materialize budget, the engine uses **per-chunk streaming GPU fold** (decode one tile → device reduce → merge) instead of a dense `vec![f32; n]`. If GPU is disabled or fails at runtime, execution falls back to CPU streaming fold (`gpu_host_materialize_exceeded` is no longer a route-time refusal when GPU features are enabled).
Constants: [`GPU_HOST_MATERIALIZE_RAM_FRACTION`](../src/query/device.rs), [`GPU_HOST_MATERIALIZE_UNKNOWN_HOST_CAP_BYTES`](../src/query/device.rs).
### Other fallback reasons
| `device_fallback_reason` | Meaning |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `gpu_feature_disabled` | Requested GPU but crate not built with `tetration-metal` / `tetration-gpu`. |
| `gpu_host_materialize_exceeded` | Dense host buffer over budget; engine may still use **streaming GPU fold** when GPU features are enabled (not an automatic CPU-only refusal). |
| `auto_below_size_threshold` | `auto` and logical selection < 64 MiB. |
| `gpu_unsupported_dtype` | Not dense **`f32`**. |
| `gpu_unsupported_op` | Tier-C / partial-axis / unsupported scalar op. |
| `tier_c_materialize` | `median` / `quantile` / `histogram` need full materialize path. |
| `preview_or_spill_only` | No `operation` in query. |
Runtime GPU failures (VRAM, decode, kernel) fall back to CPU with reasons such as `gpu_vram_exceeded`, `gpu_runtime_error`, `gpu_host_decode_failed` (see [`scalar_fold.rs`](../src/query/gpu/scalar_fold.rs)).
### Build features
```bash
cargo build --release --features tetration-metal # macOS only (Metal dep is target-gated)
cargo build --release --features tetration-gpu # Linux/Windows + NVIDIA
# tetration-rocm — same CUDA/NVRTC path today; mutually exclusive with tetration-gpu
```
Default crate features do **not** link GPU backends. CI uses platform feature sets (not `--all-features` on Linux/Windows) — see [`.github/workflows/ci.yml`](../.github/workflows/ci.yml).
### Bench expectations ([`fixtures/README.md`](../fixtures/README.md))
| Tier | Typical `device` (auto / metal on M4 Max) | Notes |
| -------------------- | ------------------------------------------ | -------------------------------------------------------------------------------------------- |
| **large** (~6.7 GiB) | `metal` when RAM allows | sum/mean/min/max on GPU; var/std on host SIMD. |
| **extra** (~20 GiB) | `cpu` or streaming GPU when built + routed | Dense path refused by host-RAM gate on most laptops; streaming GPU or CPU streaming ~2 s/op. |
Use **`mise run bench:cpu`** for apples-to-apples CPU timing; **`bench:metal`** / **`bench:gpu`** to exercise device routing.
### Device tokens (Phase 10)
| Token | Meaning |
| -------------------------------- | ---------------------------------------------------------------------------- |
| `cpu` | Host streaming fold only. |
| `auto` | Metal (macOS) or CUDA device 0 when features enabled; skips GPU < 64 MiB. |
| `metal` | Apple GPU. |
| `cuda` / `cuda:N` | NVIDIA GPU _N_. |
| `cuda:multi` | Shard chunks across all visible CUDA devices (Rayon + merge). |
| `rocm` / `rocm:N` / `rocm:multi` | ROCm build (`tetration-rocm`); same kernel path as CUDA today. |
Execution preview: `device_gpu_pipeline`, `device_gpu_multi`.
### Not implemented (Phase 10)
- GPU tier-C ops, dtypes other than **`f32`** / **`f16`** (promoted to `f32` on device).
- Meaningful speedup over CPU streaming on unified-memory Macs for full-tensor tier-A/B ops (current architecture).
## Scalability: read-many and Phase 10
**Deployment model (v1):** [README — Concurrency and scale](../README.md#concurrency-and-scale), [layout v1 — Concurrency](layout_v1.md#concurrency-informative).
```text
Write (once) → seal .tet → N readers (mmap, independent queries)
│
CPU: parallel chunk fold / linear scan (today)
GPU: dense or streaming per-chunk partials
cuda:multi / rocm:multi shard across devices
```
| Milestone | What | Unlocks |
| -------------------------- | ---------------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| **A — read-many contract** | Document + test sealed-file concurrent readers | Many CPU workers / hosts on one `.tet` without format changes |
| **B — streaming GPU fold** | Per-chunk decode + device partials ([`streaming_fold.rs`](../src/query/gpu/streaming_fold.rs)) | Large `f32` tier-A/B without full-RAM `vec![f32; n]` — **done** |
| **C — overlap I/O** | Decode ∥ GPU pipeline in [`streaming_fold.rs`](../src/query/gpu/streaming_fold.rs) | Throughput — **done** |
| **D — multi-GPU** | `cuda:multi` / `rocm:multi` chunk shards ([`multi.rs`](../src/query/gpu/multi.rs)) | Machine-scale throughput — **done** |
**A** does not require GPU. **B–D** reuse the CPU fold merge algebra on the host.
Integration test: [`src/tests/concurrent_query.rs`](../src/tests/concurrent_query.rs) (library threads + optional `tet query` process smoke).
## Module map
| Submodule | Files | Responsibility |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`plan/`** | `selection.rs`, `read_plan.rs` | JSON `selection` → global box + step; `ReadPlan` chunk I/O rows and logical geometry. |
| **`decode/`** | `chunk_decode.rs`, `indexing.rs` | Mmap slice bounds, codec decode (raw **0**, zstd **1**), scatter; row-major index ↔ coords. |
| **`materialize/`** | `mod.rs`, `parallel.rs`, `int.rs`, `stats.rs` | Full/capped materialize (all wire dtypes), export spill, parallel fill, tier-C stats. |
| **`fold/`** | `fold_policy.rs`, `linear_scan.rs`, `shared.rs`, `reduction/`, `variance_simd/`, `parallel/` (Rayon), `partial/` (`fields`, `float`, `int`), … | `FoldIoPolicy` / I/O regime; out-of-core **linear scan**; `FoldPlanOutcome`, `ReductionKind`; SIMD bulk sum/var/min-max on tier-A/B slabs (all supported float/integer tags); parallel + partial-axis streaming folds. |
| **`dispatch.rs`** | — | Dtype routing for materialize, spill, scalar/partial fold. |
| **`engine/`** | `run.rs`, `operations.rs`, `budget.rs`, … | `plan_query_*`, `build_execution_preview`, budget and spill policy. |
| **`cli/`** | `history.rs`, `info.rs`, `output/` (`mod.rs`, `plan.rs`, `quiet.rs`, `stats.rs`, …) | Platform query history JSONL; `tet info` formatters; `QueryOutputFormat`, `format_query_response`. |
Public re-exports are wired in [`engine/mod.rs`](../src/query/engine/mod.rs) and [`query/mod.rs`](../src/query/mod.rs) (`tetration::query::plan_query_empty`, `format_query_response`, `QueryOutputFormat`, `materialize_read_plan_f32_le`, `ExecutionBudget`, `spill_read_plan_f32_le`, …). Crate root exposes modules plus [`prelude`](../src/lib.rs).
## CLI query output
Library entry: **`format_query_response(&QueryResponse, QueryOutputFormat)`** — builds strings for `tet` stdout without mutating the response.
- **`full` / `json`** — serialize the full envelope (pretty vs compact). Use for debugging, golden tests, and tools that need `read_plan.chunks[]`.
- **`stats`** — [`stats.rs`](../src/query/cli/output/stats.rs) projects catalog/read*plan summaries and execution I/O + `operation*\*` fields; omits chunk rows and preview arrays.
- **`quiet`** — [`quiet.rs`](../src/query/cli/output/quiet.rs) prints a single `key=value` line; partial-axis reductions include reduced shape + sample values. Plan-only and unmatched-dataset cases still emit a short line.
Implement new presentation in **`src/query/cli/output/`**; keep planning/execution in **`engine/`**.
## Planning detail
From `QueryDocument` + catalog metadata:
1. **`plan/selection.rs`** — resolve per-axis box → `g0`, `g1_exclusive`, `step`.
2. **Chunk-touch policy** — if any `step ≠ 1`, use `strided_half_open`; else `dense_half_open_unit_step`.
3. **`catalog`** — `chunk_coords_intersecting_strided` → chunk coord list.
4. **`plan/read_plan.rs`** — `build_read_plan` → `ReadPlan` (chunk I/O rows + `logical_selection_shape`).
Each `ReadPlan.chunks` entry names one on-disk tile that intersects the selection. Chunk iteration order follows the catalog writer (last axis fastest); **decoded values** are **not** in chunk order—they are scattered into logical row-major selection order during materialization.
## Memory budget and execution strategies
Before decode, **`ExecutionBudget::resolve`** picks how much anonymous RAM the engine may use for dense in-memory paths. Precedence (highest wins):
1. Query JSON **`execution.memory_budget_bytes`**
2. Per-file **TIDX header** `memory_budget_bytes` ([`FileExecutionSettingsV1`](../src/catalog/execution.rs); see [`layout_v1.md`](layout_v1.md#chunk-index-header-32-bytes))
3. Query JSON **`execution.memory_budget_percent_bps`** (basis points; **10000 = 100%**, e.g. **2500 = 25%**)
4. Per-file TIDX **`memory_budget_percent_bps`** (**0** = engine default **25%**)
5. **256 MiB** fallback when host RAM cannot be probed
Host RAM is read best-effort via **`utils::host_memory`** (Linux `MemAvailable`, macOS free+inactive pages). Resolved budget and probe results appear on **`execution.memory_budget_bytes`**, **`execution.host_available_ram_bytes`**, **`execution.logical_selection_bytes`**, and (for **`f32`** datasets) **`execution.logical_selection_f32_bytes`** in the response.
**`build_execution_preview`** then picks a **`MemoryStrategy`** (exposed as **`execution.memory_strategy`**):
| Strategy | When chosen | Behavior |
| ---------------------------- | ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`streaming_fold`** | Tier-A/B **`operation`** (sum, mean, min, …) | Single-pass fold (scalar or partial axes); no full logical `Vec`. I/O path is chosen by [`FoldIoPolicy`](../src/query/fold/fold_policy.rs) — see [Adaptive fold I/O](#adaptive-fold-io). |
| **`in_memory_materialize`** | Tier-C **`operation`** when logical size ≤ budget | Full logical decode into RAM; op runs on buffer; preview from same buffer. |
| **`temp_spill_materialize`** | Tier-C **`operation`** when logical size > budget | Full decode to engine temp file under cache allowlist; op via mmap; file deleted when execution finishes. |
| **`mmap_spill`** | Top-level **`"spill": "path"`** and no reduction key | Full logical selection written row-major **dtype-native LE** to caller path; preview read from spilled file (one decode). |
| **`capped_in_memory`** | Preview-only (no `operation`, no spill) when logical size ≤ budget | Capped materialize into **`execution.f32_preview`** or **`execution.f64_preview`** only. |
When logical selection exceeds the budget and neither **`operation`** nor spill is requested, execution fails with a validation error suggesting a higher budget, an **`operation`**, or spill output.
Capped preview allocates only `min(cap, logical)` elements in the dtype-appropriate preview field (not the full logical tensor). Full materialize (`max_elements: None`) still requires a buffer sized to the logical selection.
### Spill path allowlist
Spill targets come from query JSON (`"spill": "…"`). **Relative paths** resolve against the **`.tet` file’s directory** (not shell cwd).
Default allowed roots (via [`SpillPathAllowlist::default_for_tet`](../src/query/engine/spill_policy.rs), applied automatically with `--tet` + `--execute`):
1. **Canonical `.tet` parent directory** — any resolved spill path **under that directory tree** is allowed (e.g. `data.tet` in `/tmp` → `/tmp/other/out.bin` is valid; use a subdirectory like `/tmp/tet_home/data.tet` if you need a narrower root).
2. **Platform cache/scratch** (best-effort, each as `…/tetration/`):
- `$XDG_CACHE_HOME/tetration` or `~/.cache/tetration` (Linux and other Unix)
- `~/.local/cache/tetration` (macOS default when `XDG_CACHE_HOME` unset; also on Linux)
- Windows: `%LOCALAPPDATA%\\tetration`
- `$TMPDIR` / `$TEMP` / `$TMP` / `tetration` subdirs when creatable
**`--spill-allow DIR`** (repeatable) **adds** extra roots to that default set.
Example relative spill beside the dataset: `"spill": "slice.bin"` → next to `data.tet`. Example cache spill: `"spill": "/home/you/.cache/tetration/job-42.bin"` (or any path under an allowed root).
### Adaptive fold I/O
Before tier-A/B fold runs, [`FoldIoPolicy::resolve`](../src/query/fold/fold_policy.rs) compares **logical selection bytes** to **~85% of probed host available RAM** (`IN_CORE_IO_HEADROOM_BPS` = 8500). The result drives chunk visit order and whether fold uses **parallel chunks**, **sequential chunks**, or **linear scan**.
```mermaid
flowchart TD
Resolve["FoldIoPolicy::resolve"]
Resolve --> Regime{"logical bytes > 85% available RAM?"}
Regime -->|no| InCore["io_regime: in_core"]
Regime -->|yes| OutCore["io_regime: out_of_core"]
InCore --> Par{"chunk_count > 1?"}
Par -->|yes| Rayon["parallel chunk fold (Rayon)"]
Par -->|no| Seq1["sequential chunk fold"]
OutCore --> Dense{"full dense raw scan + contiguous payloads?"}
Dense -->|yes| Linear["linear_scan: 64 MiB byte stream"]
Dense -->|no| Seq2["sequential or parallel chunks"]
Linear --> Read{"CLI -t path set?"}
Read -->|yes| FileRead["seek + read from file"]
Read -->|no| MmapSpan["mmap contiguous span"]
```
| Mode | When | I/O pattern |
| ------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Parallel chunk fold** | In-core, `chunk_count > 1`, not linear scan | Rayon over chunks; mmap/decode per chunk; bulk SIMD per slab when geometry allows |
| **Sequential chunk fold** | In-core single chunk, or out-of-core without linear-scan eligibility, or `execution.fold_parallel: false` | Chunks in catalog order or ascending `payload_offset` when `sequential_io` |
| **Linear scan** | Out-of-core + full dense unit-step selection + raw codec + adjacent payloads summing to logical size | One contiguous byte span; 64 MiB windows; [`push_f32_le_bytes`](../src/query/fold/reduction/value_accum.rs) SIMD per window |
**Linear scan eligibility:** [`detect_contiguous_raw_span`](../src/query/fold/linear_scan.rs) requires every touched chunk to use **raw codec 0**, payloads **abutting in plan order**, and total raw bytes = logical selection size. Converted multi-chunk **`f32`** grids from `write_raw_array_file` typically satisfy this; zstd chunks or strided partial selections do not.
**Query hint:** `"execution": { "fold_parallel": false }` forces **sequential chunk visits** (offset order when full dense); it does **not** enable linear scan. `"fold_parallel": true` on an out-of-core linear-scan query keeps parallel chunk fold instead of linear scan.
**Execution stats** (also in **`--format stats`**): `io_regime` (`in_core` / `out_of_core` / `unknown`), `fold_parallel`, `fold_linear_scan`.
### Streaming fold performance
**Behavior:** [`streaming_fold`](../src/query/engine/budget.rs) tier-A/B ops route through [`scalar_fold`](../src/query/dispatch.rs). When [`use_parallel_fold`](../src/query/fold/parallel/mod.rs) is true, fold uses Rayon over disjoint chunks (partial [`ValueAccum`](../src/query/fold/reduction/value_accum.rs) + `merge_from`). When **`fold_linear_scan`** is true, fold uses [`fold_read_plan_scalar_linear`](../src/query/fold/linear_scan.rs) instead — single-threaded byte stream, no per-chunk mmap fan-out.
So a full-file scalar op still **reads every payload byte once** and **touches every element** for the accumulator. Wall time is dominated by **page-cache / disk bandwidth** and CPU over the selection when out-of-core, or **memory bandwidth + cores** when in-core.
**SIMD bulk fold (tier-A/B, `f32` slabs):** [`variance_simd`](../src/query/fold/variance_simd/mod.rs) — NEON on aarch64, SSE2 on x86_64, scalar fallback:
- **mean / sum / var / std** — single-pass sum + sum-of-squares per slab → Welford merge
- **min / max** — single-pass vector min/max per slab → accumulator merge
Used in both parallel chunk fold (dense 1D tile bulk path) and linear scan windows.
**Reference (local, gitignored fixture):** `fixtures/extra_large/h5/tensor_20gb.tet` — 20 GiB logical **`f32`**, 320 chunks × 64 MiB raw (`dataset`: **`data`**). After convert from `tensor_20gb.h5`:
```bash
echo '{"dataset":"data","layout_version":1,"mean":[]}' \
| tet query -t fixtures/extra_large/h5/tensor_20gb.tet -x -q
```
**In-core** (data fits ~85% of available RAM — e.g. Mac Studio with ~25 GiB free): expect on the order of **~0.5–0.6 s** for full-tensor **mean** and tier-A/B ops over 20 GiB (~30–40 GiB/s effective from warm page cache). Response checks:
- **`execution.memory_strategy`**: `"streaming_fold"`
- **`execution.io_regime`**: `"in_core"`
- **`execution.fold_parallel`**: `true`
- **`execution.fold_linear_scan`**: `false`
- **`read_plan.chunk_count`**: `320`
**Out-of-core** (logical size exceeds headroom — e.g. 20 GiB on a 16 GiB machine with ~6 GiB free): linear scan when payloads are contiguous. Warm 2nd pass on Apple Silicon (May 2026): **~3.7–4.0 s** for mean/sum/std/var/min/max (~2× warm HDF5 on the same fixture). Response checks:
- **`execution.io_regime`**: `"out_of_core"`
- **`execution.fold_linear_scan`**: `true`
- **`execution.fold_parallel`**: `false`
See [`fixtures/bench_results/latest.md`](../fixtures/bench_results/latest.md) for archived runs (`mise run bench:h5`).
A **single-chunk** slice (`selection` stopping after 16 777 216 elements) completes quickly — same fold path, one chunk.
**Parallel vs linear:** per-chunk parallel fold wins when the working set fits page cache. Linear scan wins when parallel mmap/page faults thrash on cold or oversubscribed RAM but payloads are one contiguous raw hyperslab on disk.
## Materialization and operations
```mermaid
flowchart TD
BEP["build_execution_preview"]
Budget["ExecutionBudget::resolve"]
BEP --> Budget
Budget --> OpGate{"operation set?"}
OpGate -->|yes| TierGate{"materialize tier?"}
TierGate -->|streaming| FoldPath["streaming_fold"]
FoldPath --> Axes{"axes empty?"}
Axes -->|yes| ScalarFold["fold_read_plan_scalar_operation"]
Axes -->|no| PartialFold["fold_read_plan_partial_operation"]
ScalarFold --> OpFields["operation_*"]
PartialFold --> Reduced["operation_reduced_*"]
OpFields --> Done["QueryExecutionPreview"]
Reduced --> Done
TierGate -->|materialize-required| MatGate{"logical ≤ budget?"}
MatGate -->|yes| InRam["in_memory_materialize"]
MatGate -->|no| TempSpill["temp_spill_materialize → delete on drop"]
InRam --> TierCOp["median, quantile, histogram"]
TempSpill --> TierCOp
TierCOp --> Done
OpGate -->|no| SpillGate{"spill set?"}
SpillGate -->|yes| Spill["spill_read_plan_*_le → mmap_spill"]
Spill --> Done
SpillGate -->|no| CapMat["capped_in_memory materialize"]
CapMat --> Preview["f32_preview / f64_preview"]
Preview --> Done
```
- **`materialize_read_plan_f32_le`** / **`materialize_read_plan_f64_le`** — full logical tensor (caller must size for `logical_f32_element_count` and element width).
- **`materialize_read_plan_f32_le_into`** / **`materialize_read_plan_f64_le_into`** — same decode path into a caller-owned buffer (no `Vec` allocation for the output).
- **`materialize_read_plan_f32_le_parallel`** / **`materialize_read_plan_f64_le_parallel`** and **`_into_parallel`** twins — Rayon over planned chunks (raw and zstd). **`build_execution_preview`** uses parallel decode when the read plan touches more than one chunk and materialization is required.
- **`planned_chunk_mmap_slices`** — zero-copy raw codec slices only (no zstd).
- **`spill_read_plan_f32_le`** / **`spill_read_plan_f64_le`** — full logical decode to a caller-owned file path (used by **`mmap_spill`** strategy).
## `QueryResponse` fields (engine-produced)
| Field | When set |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `catalog` | Always with `--tet` / `plan_query_with_tet_mmap`. |
| `catalog.file_execution` | Matched dataset; mirrors TIDX header execution settings. |
| `catalog.dataset_f32_bytes` | Matched **`f32`** dataset; `4 × product(shape)`. |
| `catalog.dataset_f64_bytes` | Matched **`f64`** dataset; `8 × product(shape)`. |
| `read_plan` | Dataset matched; lists touched chunks and selection geometry. |
| `execution` | `raw_f32_preview_max` is `Some(n)` (including `n = 0` when an **`operation`** or **`spill`** is set). |
| `execution.f32_preview` | First `n` logical row-major floats (`n = 0` → empty vec); **`f32`** datasets only. |
| `execution.f64_preview` | First `n` logical row-major doubles (`n = 0` → empty vec); **`f64`** datasets only. |
| `execution.operation_*` | Scalar aggregates over full logical selection; preview cap does not truncate them. |
| `execution.operation_reduced_*` | Partial-axis reductions + `operation_reduced_shape`. |
| `execution.memory_strategy` | `streaming_fold`, `capped_in_memory`, `mmap_spill`, `in_memory_materialize`, `temp_spill_materialize`. |
| `execution.io_regime` | `in_core`, `out_of_core`, or `unknown` (streaming fold only). |
| `execution.fold_parallel` | Whether parallel chunk fold was selected (`streaming_fold`). |
| `execution.fold_linear_scan` | Whether out-of-core linear byte-stream fold ran (`streaming_fold`). |
| `execution.memory_budget_bytes` | Resolved RAM budget used for this run. |
| `execution.host_available_ram_bytes` | Host probe at budget resolution, when available. |
| `execution.logical_selection_f32_bytes` | `4 × logical_f32_element_count` (f32 datasets). |
| `execution.logical_selection_bytes` | `elem_size × logical_f32_element_count` for the dataset dtype. |
| `execution.spill_f32_path` / `spill_f32_bytes` | After **`mmap_spill`** decode. |
## Chunk-touch policy strings
Stable tokens on `ReadPlan.chunk_touch_policy` (see [`CHUNK_TOUCH_POLICY`](../src/query/types/plan.rs)):
- **`dense_half_open_unit_step`** — JSON `step` omitted or 1; chunk list follows dense half-open intervals.
- **`strided_half_open`** — per-axis JSON `step` affects which chunks are touched.
## Related docs
- On-disk layout (TIDX execution header): [`layout_v1.md`](layout_v1.md)
- Axis metadata (dimension names vs coordinates): [`layout_v1.md` — axis metadata](layout_v1.md#axis-metadata-phase-7-baseline)
- Roadmap summary: [`README.md`](../README.md#library-use)
## Dimension names vs coordinate labels
Each **query document** names **one dataset** in the `.tet` file (`"dataset": "temperature"`); multi-dataset join/group is out of scope (see [Tier 3](#tier-3--not-operation-enum-variants)). Decimal axis indices remain the internal wire form; footer metadata ([`layout_v1.md — axis metadata`](layout_v1.md#axis-metadata-phase-7-baseline)) enables two user-facing layers:
### Dimension names (axis names)
- **One string per axis** — the name of the dimension, not each slot along it.
- **Query use (shipped):** resolve `"mean": "time"` → internal axis `0` at plan time when footer `dim_names` is set; decimal indices and numeric wire forms still accepted.
- **Size:** O(`ndim`) — fits in dataset attrs or file header.
### Coordinate labels (index values)
- **One value per position** along an axis — e.g. every row’s timestamp, every column’s variable name, every station id.
- **Query use (shipped):** `selection[].start_label` / `stop_label` resolve to axis indices at plan time when footer `coords` exist (axis key = `dim_names[d]` or `"d"`). Filter-by-value and group-by remain later.
- **Size:** O(`shape[d]`) — may be stored inline, as attrs, or as a separate 1D array/dataset in the file.
### What labels do _not_ imply
| Capability | Needs |
| ------------------------------- | ------------------------------------------------------------------- |
| `"mean": "time"` (axis name) | **Dimension names** only |
| Slice rows 100–200 by index | **Today** — numeric `selection` |
| Slice by timestamp / station id | **Coordinate labels** on `selection` (**done**); range/filter later |
| Fast lookup without full scan | Coords **+ optional index** (sorted array, hash, …) |
| `GROUP BY station` | Coords on key axis **+ group-by op** |
| SQL-style join two datasets | Two queries or spill + caller; **non-goal** as full engine |
**Append vs join:** growing one dataset over time is a **writer** concern (Phase 7 session writer). Combining two datasets on shared coordinates is **join** semantics — metadata helps **alignment checks**, not automatic join in the query engine.
## Query document (JSON and TOML)
The wire format is **flat**: one top-level reduction key per document (`mean`, `sum`, …). Nested `"operation": { … }` is **rejected** at parse time. **TOML** uses the same keys and semantics after conversion to JSON (see examples below).
### JSON examples
| Write | Meaning (internal `operation.axes`) |
| --------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `"mean": []` | Scalar mean over all selected elements |
| `"mean": 0` | Mean over axis **0** (`["0"]`) |
| `"sum": [0, 1]` | Partial reduction over axes 0 and 1 |
| `"quantile": { "q": 0.95, "axis": 0 }` | Quantile on axis 0 (or `"axes": [0, 1]` for multi-axis) |
| `"histogram": { "bins": 10, "axis": 0 }` | Histogram on axis 0 |
| `"histogram": { "bins": 10, "min": 0, "max": 1 }` | Histogram with fixed edge range (both `min` and `max` required) |
| `"nan_count": []` / `"nan_count": 0` | Count of NaN elements (scalar / partial) |
| `"inf_count": []` / `"inf_count": 0` | Count of ±infinity elements (scalar / partial; integers contribute 0) |
| `"null_count": []` or `{ "fill": 99, "axis": 0 }` | Count of fill-missing values (fill from query or dataset attrs) |
| `"selection": [{ "start_label": "r0", "stop_label": "r1" }, …]` | Half-open slice by coordinate label (requires footer `coords`) |
| `"covariance": { "axis": 0 }` / `"correlation": 0` | Rank-2 only; `axis` = observation dimension; `operation_*_order` = variable count |
| `"execution": { "memory_budget_percent": 40 }` | **40%** of host RAM (`memory_budget_percent_bps` = 4000 internally) |
| `"execution": { "fold_parallel": false }` | Force sequential chunk visits (not linear scan); default is policy-driven |
| `"spill": "slice.bin"` | Full logical tensor export beside the `.tet` parent (allowlist applies) |
`memory_budget_percent_bps` is still accepted in JSON for embedders; serialization prefers **`memory_budget_percent`**. Nested `"output": { … }` is rejected (use **`spill`**). Axis specs accept **non-negative decimals** (`0`) or **dimension names** (`"time"`) when `dim_names` is in footer metadata — see [Dimension names vs coordinate labels](#dimension-names-vs-coordinate-labels).
At most **one** reduction key per document. Unknown top-level fields are rejected (`deny_unknown_fields` style in [`parse_query_value`](../src/query/document_wire.rs)).
### TOML examples
Equivalent TOML for the same wire (parametric ops use inline tables; `selection` uses `[[selection]]` array-of-tables):
| TOML | Same as JSON |
| --------------------------------------------------------- | ------------------------------------------------------- |
| `dataset = "temperature"` / `mean = []` | `"dataset": "temperature", "mean": []` |
| `mean = 0` | `"mean": 0` |
| `sum = [0, 1]` | `"sum": [0, 1]` |
| `[quantile]` / `q = 0.95` / `axis = 0` | `"quantile": { "q": 0.95, "axis": 0 }` |
| `[execution]` / `memory_budget_percent = 40` | `"execution": { "memory_budget_percent": 40 }` |
| `spill = "slice.bin"` | `"spill": "slice.bin"` |
| `[[selection]]` / `start = 0` / `stop = 100` / `step = 2` | `"selection": [{ "start": 0, "stop": 100, "step": 2 }]` |
```toml
dataset = "temperature"
mean = []
[[selection]]
start = 0
stop = 100
step = 2
```
```bash
tet query q.toml -t data.tet -x -q
```
**Not supported in TOML:** nested `"operation"` / `"output"` objects (same rejection as JSON). Line-oriented query profiles remain a possible later front-end.
## Operations (shipped in v1)
Reduction keys map to response fields below. **Scalar** = empty axis list (`[]` on the wire); **partial** = one or more axis indices.
| Operation / JSON key | Scalar (`[]`) | Non-empty axes (partial reduction) |
| -------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `sum` | `operation_sum` | `operation_reduced_sum` + `operation_reduced_shape` |
| `mean` | `operation_mean` | `operation_reduced_mean` + `operation_reduced_shape` |
| `min` | `operation_min` | `operation_reduced_min` + `operation_reduced_shape` |
| `max` | `operation_max` | `operation_reduced_max` + `operation_reduced_shape` |
| `count` | `operation_element_count` | `operation_reduced_count` + `operation_reduced_shape` |
| `var` | `operation_var` | `operation_reduced_var` + `operation_reduced_shape` |
| `std` | `operation_std` | `operation_reduced_std` + `operation_reduced_shape` |
| `product` | `operation_product` | `operation_reduced_product` + `operation_reduced_shape` |
| `norm_l1` | `operation_norm_l1` | `operation_reduced_norm_l1` + `operation_reduced_shape` |
| `norm_l2` | `operation_norm_l2` | `operation_reduced_norm_l2` + `operation_reduced_shape` |
| `all_finite` | `operation_all_finite` | `operation_reduced_all_finite` + `operation_reduced_shape` |
| `any_nan` | `operation_any_nan` | `operation_reduced_any_nan` + `operation_reduced_shape` |
| `arg_min` | `operation_argmin_index` | `operation_reduced_argmin` + `operation_reduced_shape` |
| `arg_max` | `operation_argmax_index` | `operation_reduced_argmax` + `operation_reduced_shape` |
| `median` | `operation_median` | `operation_reduced_median` + `operation_reduced_shape` |
| `quantile` | `operation_quantile` (`q` field) | `operation_reduced_quantile` + `operation_reduced_shape` |
| `histogram` | `operation_histogram_counts`, `operation_histogram_edges` | `operation_reduced_histogram_counts` (flat `out_len × bins`, row-major; edges omitted) + `operation_reduced_shape` |
Population **`var` / `std`**, `ddof = 0`. **`norm_l2`** is √(sum of squares).
### Dtypes
| Wire tag | Name | Writer | Query execution |
| -------- | ----- | ------ | ----------------------------------------------- |
| `1` | `f32` | yes | yes (preview in `f32_preview`) |
| `2` | `f64` | yes | yes (preview in `f64_preview`) |
| `3` | `i32` | yes | yes (aggregates promote to `f64` in fold paths) |
| `4` | `i64` | yes | yes (aggregates promote to `f64` in fold paths) |
Element size and budget math: [`ElementDtype`](../src/utils/dtype.rs).
Quantile example (exact selection on sorted values, linear blend between adjacent ranks):
```json
{ "dataset": "a", "quantile": { "q": 0.95 } }
```
Histogram example (equal-width bins from per-cell min/max; scalar returns edges):
```json
{ "dataset": "a", "histogram": { "axis": 0, "bins": 10 } }
```
Example:
```json
{
"dataset": "temperature",
"mean": 0,
"execution": { "memory_budget_percent": 40 }
}
```
Spill example (full logical tensor to disk, no JSON preview floats required):
```json
{
"dataset": "temperature",
"spill": "/tmp/temp_slice.bin"
}
```
**Execution paths today:**
- **Preview only** (no `operation`, no spill) — **`capped_in_memory`** when logical size ≤ budget.
- **Streaming ops** (`sum`, `mean`, …) — **`streaming_fold`**: scalar (empty axis list) via **`fold_read_plan_scalar_operation`**; partial axes via **`fold_read_plan_partial_operation`**. Preview is the first `n` logical values in dtype-matched preview arrays (`f32_preview`, `u8_preview`, …) when `raw_f32_preview_max > 0`.
- **Materialize-required ops** (`median`, `quantile`, `histogram`) — **`in_memory_materialize`** when logical size ≤ budget; **`temp_spill_materialize`** when over budget (engine temp file under cache allowlist, removed after the op). Requires `--tet` (or explicit **`SpillPathAllowlist`**) so temp paths are allowed.
- **Export spill** — **`mmap_spill`** when **`spill`** is set and no reduction key. Preview is read from the spilled file (single full decode; dtype-native bytes).
Supported dtypes: wire tags in [`DATASET_DTYPE_TAG_V1`](../src/catalog/mod.rs) — `f32` = 1, `f64` = 2, `i32` = 3, `i64` = 4, `u8` = 5, `u16` = 6, `i16` = 7, `u32` = 8, `f16` = 9, `u64` = 10 (see [`layout_v1.md`](layout_v1.md)). Tier-A/B aggregates promote integers to **`f64`** in `operation_*`. The preview cap does **not** truncate `operation_*` aggregates. **`f16` partial-axis** fold is still rejected at dispatch (scalar tier-A/B only).
## Operations roadmap (planned)
New ops should declare which **implementation tier** they use. That keeps “huge tensor + one number” fast while harder stats stay explicit about memory.
### Implementation tiers
| Tier | Name | When to use | Engine pattern |
| ----- | -------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| **A** | **Scalar fold** | `axes: []`, associative or online stats | [`scalar_fold`](../src/query/dispatch.rs) → parallel chunks (in-core), linear scan (out-of-core contiguous raw), or sequential chunks. |
| **B** | **Partial-axis fold** | Non-empty `axes`, element-wise combine along dimensions | [`partial`](../src/query/fold/partial/mod.rs) with same `FoldIoPolicy` chunk ordering. |
| **C** | **Materialize-required** | Needs full logical tensor order, sort, or index of extrema | Full decode (or spill file); may add new `operation_*` response fields. |
| **D** | **Out of scope for `Operation`** | Writers, dtype views, foreign format import | Separate APIs (`materialize_*`, `tet convert`, metadata), not the JSON `operation` enum. |
### Tier 1 — shipped (v1)
**Done:** tier-A/B streaming ops (`sum`, `mean`, `min`, `max`, `count`, `var`, `std`, `product`, `norm_l1`, `norm_l2`, `all_finite`, `any_nan`, `arg_min`, `arg_max`) — scalar + partial axes.
### Tier 2 — tensor stats (shipped)
**Shipped (index ops):** `arg_min`, `arg_max` (scalar logical index; partial = index within reduced axes fiber).
| Op | Tier (typical) | Notes |
| ---------------------------- | -------------- | ------------------------------------------------------------------------ |
| **`argmin` / `argmax`** | C → **A/B** | **Done** as `arg_min` / `arg_max`. |
| **`median`** | C | **Done** (scalar + partial axes). |
| **`quantile` / `histogram`** | C | **Done** (scalar + partial axes; histogram partial returns counts only). |
### Phase 9 ops (shipped)
See [README — Library use](../README.md#library-use) and Phase 9 notes in this file.
| Op (wire key) | Tier (typical) | Notes |
| ---------------------------- | -------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| **`nan_count`** | A/B | Count of NaN elements; complements **`any_nan`** (boolean). |
| **`inf_count`** | A/B | Count of ±infinity; complements **`all_finite`** (boolean). |
| **`null_count`** | A/B | Fill from query `fill` or footer attrs (`_FillValue`, `missing_value`, `fill_value`). |
| **Related QC counts** | A/B | Deferred — e.g. `finite_count`. |
| **Histogram `min` / `max`** | C | **Done** — caller-supplied bin edges on scalar histogram (both required when either set). |
| **Covariance / correlation** | C | **Done** — rank-2; `axis` = observation dimension (`operation_covariance` / `operation_correlation` row-major `order×order`). |
| **Named axis in JSON** | — | `"mean": "time"` via `dim_names` (resolver only). |
| **Coord label slice** | — | **Done** — `selection` `start_label` / `stop_label` ([`resolve_selection.rs`](../src/query/resolve_selection.rs)). |
### Tier 3 — not `Operation` enum variants
These match the product vision but belong **beside** the reduction enum:
| Capability | Why separate |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Read / export** | Plan + materialize or `output.spill` ([`OutputHint::SpillArray`](../src/query/types/document.rs)); **`.tet` → Zarr v3** via `tet export` ([`export`](../src/export/mod.rs)). |
| **`cast` / integer dtypes** | **`i32` / `i64`** on disk and in materialize/query fold (**done**). |
| **Named axis labels** | **Done** — footer `dim_names` resolves `"time"` to axis index at plan time ([`resolve_axes.rs`](../src/query/resolve_axes.rs)). |
| **Coordinate labels / filter-by-value** | **Slice by label** shipped (`start_label` / `stop_label`); filter-by-value / group-by and optional lookup index remain later. |
| **`rechunk` / resample** | Writer / transform path, not read-time aggregate. |
| **Linear algebra** (`matmul`, `einsum`) | Belongs in caller libraries on materialized slabs. |
| **Spectral / ML** (FFT, CWT, conv, train) | Same: materialize or spill, then NumPy / SciPy / PyTorch / JAX — not chunk-local in the engine. |
| **SQL / joins** | Explicit non-goal (see [README](../README.md)). |
| **CLI query history** | **Done** — platform cache JSONL (`tet qhist list` / `run`; `hist` alias); `TET_QUERY_HISTORY_MAX` rotation; `.tet` footer via `tet info --history`. |
### Non-goals for the JSON `operation` field
- Arbitrary per-chunk user callbacks (needs a sandbox and stable ABI).
- Plugin codecs or filters beyond the v1 catalog codec tags.
- Guarantees about numerical order beyond logical row-major **preview** order (aggregates are commutative where noted).
- **FFT, CWT, convolution, and ML ops** — export a hyperslab, then run ecosystem libraries (future Python bindings repo — see [`ffi.md`](ffi.md)).
- **Query replay / result cache in `.tet`** — use optional client-side memoization (`tet qhist` stores recent query JSON in platform cache only; does not mutate the file or skip decode by default).
When adding an op, update this table, [`Operation`](../src/query/types/document.rs), `validate_query` / `document.rs`, `reduction/` / `operations.rs` / `partial/`, and (if tier **C**) `materialize_stats.rs`.
## JSON security (input and output)
The query control plane (JSON or TOML input) is a **declarative document**, not executable code. There is no query language interpreter, SQL engine, or “run this string from the payload” path. Security still matters in **both directions**: untrusted input must not drive memory-unsafe or out-of-policy behavior, and untrusted consumers must not treat output JSON as shell/HTML/SQL without encoding.
### Threat model (v1)
| Surface | Source | Risk if mishandled |
| ---------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| **Query JSON/TOML in** | User, HTTP body, stdin, agent prompt | DoS (huge/deep payload), logic abuse (absurd selection), confused deputy (if a host passes paths from the document) |
| **`.tet` mmap** | Caller-chosen file path (CLI flag, not JSON field today) | Malicious file → bad index spans (mitigated in [catalog robustness](#robustness-catalog-index)) |
| **Query JSON out** | `QueryResponse` pretty-print | Log/UI injection if embedded raw; unsafe `eval` in downstream scripts |
| **Binary payloads** | Chunk bytes on disk | Not inlined in JSON; decoded only through catalog + read plan |
| **Spill path** | `"spill"` string in query JSON/TOML | Host path chosen by caller; validated against [`SpillPathAllowlist`](../src/query/engine/spill_policy.rs) |
The **dataset name** and **operation axis labels** in JSON are echoed in responses and errors. Treat them as **untrusted display data** unless your deployment pre-validates them.
### Input protections (today)
Implemented in [`document.rs`](../src/query/document.rs) and planning:
1. **Typed `serde` deserialization** — JSON maps into fixed structs/enums ([`QueryDocument`](../src/query/types/document.rs), [`Operation`](../src/query/types/document.rs)); there is no dynamic “operation name → callback” table fed by arbitrary strings.
2. **Closed `operation` enum** — Only the v1 ops listed above deserialize; unknown tags fail at parse time.
3. **Axis tokens restricted** — `operation.axes` entries must be non-empty **ASCII decimal** indices (e.g. `"0"`) or **dimension names** (e.g. `"time"`); arbitrary expressions and coordinate values are rejected ([`validate_operation_axis_token`](../src/query/document.rs)).
4. **Slice sanity** — `step != 0`; when both `start` and `stop` are set, `start < stop`.
5. **Non-empty `dataset`** — Whitespace-only names rejected.
6. **Catalog binding** — After parse, selection rank/shape and axis indices are checked against the mmap’d dataset ([`plan_query_with_tet_mmap`](../src/query/engine/run.rs), [`resolved_dense_global_box`](../src/query/plan/selection.rs)); JSON cannot name chunk file offsets directly.
7. **No `.tet` path in query JSON** — Opening a `.tet` file uses the **host** path (`--tet`, API argument), not a field inside the query document. Spill **output** paths _are_ in JSON today; deployments should allowlist or rewrite them.
8. **Size and shape caps** — [`parse_query_json`](../src/query/document.rs) / [`parse_query_toml`](../src/query/document_toml.rs) reject payloads over [`QueryLimits::DEFAULT.max_json_bytes`](../src/query/document.rs) (1 MiB) and nesting depth over [`QueryLimits::DEFAULT.max_json_depth`](../src/query/document.rs) (64) on the converted JSON tree. [`validate_query`](../src/query/document.rs) caps `dataset` length, `selection` rank and `operation.axes` count (≤ [`MAX_NDIM`](../src/catalog/mod.rs)), and per-axis label length via **`QueryLimits::DEFAULT`**.
9. **`deny_unknown_fields`** — [`QueryDocument`](../src/query/types/document.rs) and nested input types reject unexpected keys at parse time (after TOML→JSON conversion).
10. **Fuzz / property tests** — `src/tests/query.rs` proptest: random UTF-8 must not panic in `parse_query_json` / `validate_query`; TOML covered by unit tests on representative profiles.
**Not enforced yet (deployments should add limits):**
- Canonical JSON / duplicate-key rejection (depends on `serde_json` behavior).
- Rate limiting and authentication on any HTTP wrapper around `tet query`.
- Spill path allowlist / sandbox directory policy.
### Output protections (today)
1. **`serde_json` serialization** — Strings in `QueryResponse` are JSON-escaped when written (`tet query --format full` pretty-print, or `json` / `stats` compact JSON).
2. **Bounded preview arrays** — `execution.f32_preview` / `execution.f64_preview` are capped by `--preview-f32` / `raw_f32_preview_max`; aggregates use full-tensor math but return numeric summaries, not opaque blobs.
3. **Server-generated messages** — Most `message` text is produced by the engine; user strings appear mainly as echoed `dataset` / axis labels and in validation errors.
### Caller responsibilities (either direction)
#### Ingesting query JSON or TOML
- Cap input size and parse time; reject documents above policy limits before calling the library.
- Do not build shell commands, SQL, or file paths by string-concatenating raw document fields.
- Keep the `.tet` path under caller control (allowlist directories, no user-supplied absolute paths in multi-tenant services unless intended).
- For **`spill`**, validate or rewrite the path before execution (path traversal, writable location, quota).
**Consuming `QueryResponse` JSON**
- Treat the document as **data**, not instructions: never `eval`, `source`, or template-interpolate response JSON into code without a schema.
- When embedding in HTML, email, or terminals, apply normal contextual escaping (JSON ≠ safe HTML).
- For logs, prefer structured JSON logging or strip/control characters in echoed `dataset` names if logs are human-facing.
### Hardening roadmap
| Item | Direction | Notes |
| ------------------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------- |
| Input size / depth limits | In | **Done** — `QueryLimits::DEFAULT` |
| `deny_unknown_fields` | In | **Done** on query input types |
| Dataset / axis length caps | In | **Done** in `validate_query` via `QueryLimits` |
| Spill path allowlist | In/out | **Done** — `SpillPathAllowlist`, `plan_query_with_tet_mmap_ex`, CLI `--spill-allow` |
| Response schema version + stability | Out | Document breaking changes |
| Fuzz `parse_query_json` / `validate_query` | In | **Basic** proptest in `src/tests/query.rs`; TOML smoke in same module |
| TOML query front-end | In | **Done** — [`parse_query_toml`](../src/query/document_toml.rs), CLI `.toml` + auto-detect |
| Redaction mode for echoed fields | Out | Multi-tenant logging |
| Capped preview without full-buffer alloc | In | **Done** — bounded scatter buffer when `max_elements < logical` |
| Parallel streaming fold (tier A/B ops) | In | **Done** — Rayon over chunks when in-core; see [Adaptive fold I/O](#adaptive-fold-io) |
| Out-of-core linear scan | In | **Done** — contiguous raw hyperslab + SIMD windows; [PR #7](https://github.com/Latka-Industries/tetration/pull/7) |
## Robustness (catalog index)
- Catalog robustness and index property tests: [`src/tests/catalog.rs`](../src/tests/catalog.rs).
- Query planning, materialize, operations, memory budget, all dtype paths, tier-C stats: [`src/tests/query.rs`](../src/tests/query.rs).
- Fold I/O policy and contiguous-span detection: [`src/tests/fold.rs`](../src/tests/fold.rs).
- SIMD bulk folds vs scalar reference: [`src/tests/variance_simd.rs`](../src/tests/variance_simd.rs).
- Bulk variance vs Welford reference: [`src/tests/reduction.rs`](../src/tests/reduction.rs).
- File health: [`src/verify/`](../src/verify/), [`src/tests/verify.rs`](../src/tests/verify.rs), [`src/tests/repair.rs`](../src/tests/repair.rs), [`src/tests/verify_fixtures.rs`](../src/tests/verify_fixtures.rs), committed CLI smoke [`fixtures/small/tet/`](../fixtures/small/tet/README.md).
- Phase 9 ops + export: [`src/tests/resolve_axes.rs`](../src/tests/resolve_axes.rs), [`resolve_selection.rs`](../src/tests/resolve_selection.rs), [`covariance.rs`](../src/tests/covariance.rs), [`export.rs`](../src/tests/export.rs).
- Payload decode uses [`src/utils/le_pod.rs`](../src/utils/le_pod.rs) (`f32_le` / `f64_le` / …; bytemuck; aligned cast when possible).
## Intentional gaps (v1)
- Direct callers can still use **`materialize_read_plan_f32_le`** / **`_f64_le`** (always sequential) or **`_parallel`** twins; execution picks parallel decode/fold per `FoldIoPolicy` (see [Adaptive fold I/O](#adaptive-fold-io)).
- Filter-by-value and **group-by** on coordinate labels remain deferred; **`operation.axes`** accepts decimal indices or **dimension names** (`dim_names`), not coordinate values — see [Dimension names vs coordinate labels](#dimension-names-vs-coordinate-labels).
- Spill path must lie under default roots (canonical **`.tet` parent** and descendants + platform `…/tetration` cache dirs) or `--spill-allow`; relative **`"spill"`** paths resolve against the **`.tet` directory** (not shell cwd).
- Integer and narrow float dtypes (`u8`–`u64`, `f16`) are supported in writers, convert, and query execution (tier-A/B aggregates promote to `f64` for fold paths).
- Query hardening (JSON and TOML): byte size, depth, `deny_unknown_fields`, and string/rank caps via **`QueryLimits`**; spill path policy via **`SpillPathAllowlist`** (see [JSON security](#json-security-input-and-output)).