# Offline tool-output measurement
[Feature docs index](README.md) · [Repository README](../../README.md)
## Purpose
Estimate potential tool-result token savings from stored sessions without changing live provider requests.
## Run
Run the exploratory measurement prototype explicitly with:
```text
magi-code sessions measure-compression --model <MODEL>
```
It reads only active primary JSONL files directly under `sessions/` and active
subagent JSONL files directly under `sessions/subagents/`. Discovery traverses
the absolute session path component by component from a stable filesystem
anchor without following ancestor or root symlinks, validates each held
directory handle, opens `subagents/` without following a symlink, enumerates
both directories through held directory capabilities, and opens each candidate
read-only relative to the matching directory capability. Selected files are
not reopened through ambient path access.
Only active-name regular files with stable identity metadata and no more than
the internal 64 MiB per-file cap become eligible. Normal archives, sidecars,
temporary files, history directories, and other non-candidates are ignored
without skipped-input counts. Symlinked active JSONL candidates, unsafe active
file types, and an unsafe `subagents/` path are rejected and counted as
skipped. The command emits one aggregate JSON report with counts and local
token estimates; it does not emit session paths, IDs, raw output, excerpts, or
file contents. The report is privacy-oriented and read-only.
Only supported `openai-codex` text model families are accepted; embedding
models are not. The `o200k_base` family accepts exact `gpt-5`, `gpt-4.1`,
`gpt-4o`, `gpt-4.5`, `o1`, `o3`, and `o4` families with either the exact
family name or a valid `-` suffix; `gpt-5` also accepts version suffixes that
begin with `.`. `codex-` models require a nonempty valid identifier after the
prefix. The `cl100k_base` family accepts exact `gpt-4` and `gpt-3.5-turbo`
families with the same valid `-` suffix rule. Known non-text modality
identifiers (`embedding`, `audio`, `realtime`, `transcribe`, `tts`, and
`image`) are rejected. These are local estimates, not provider billing
counts. An unsupported model is rejected without scanning.
Optional `--max-files` and `--max-bytes` flags bound sampling. `--max-files`
defaults to 100 and has a maximum of 10,000. `--max-bytes` is a byte count;
it defaults to 128 MiB and has a maximum of 1 GiB (with a 1 MiB minimum).
This prototype does not change provider-visible output, sessions, settings,
authentication, or files under `MC_HOME`. It does not use the network, enable
any production codec, or implement TOON.
### Measurement report
The JSON report is aggregate-only. Its fields are:
| `prototype` | Always `offline_compression_measurement_v0`; identifies this exploratory report, not a production codec or setting. |
| `schema_version`, `model`, `encoding_family`, `scope`, `exploratory` | Report version, requested model, selected local BPE family, fixed active-session scope, and the fixed exploratory marker. |
| `measurement_unit` | Always `unique_stored_tool_results`; measurements are reported per stored tool result rather than per provider request. |
| `request_weighted` | Always `false`; results are not weighted by request volume. |
| `replay_compaction_applied` | Always `false`; no replay compaction is applied before measurement. |
| `sampling` | File and byte sampling totals. Eligible files are opened and measured from held read-only handles; rejected unsafe files and files over the internal 64 MiB per-file cap are not part of these totals. |
| `files.considered` | Eligible active regular JSONL candidates after safe opening and the per-file cap, before sampling. `scanned` is the number selected and whose handles passed the scan checks; `sampled_out` covers eligible files omitted by the file/byte limits. `skipped` covers rejected active candidates such as symlinks; `unreadable`, `unstable`, and `limit_hit` are separate safety/limit counts. `limit_hit` includes file-level input, result, candidate, and per-file bounds. |
| `jsonl.lines`, `valid_events`, `malformed`, `oversized`, `limit_hit` | Bounded JSONL scan counts. `oversized` counts lines over the line cap; `limit_hit` counts files that reached the line-count cap. |
| `tool_results.found`, `empty`, `measured`, `legacy`, `invalid`, `skipped_limit` | Tool-result counts. Empty results are found and measured when their complete item framing is available, but normally have no codec candidate. `invalid` counts found tool-result events that fail required-field or framing validation; `skipped_limit` counts only known tool results over the candidate-size cap. |
| `baseline` | Raw content bytes and Codex Responses-style local BPE projection tokens for each `function_call_output` item before a codec; these are not full provider-context or billing-token counts. |
| `codecs` | Per-codec attempts, validated candidates, strict improvements, selections, candidate token totals, and selected savings. |
| `portfolio` | Tokens selected after independent per-result selection, saved tokens, transformed output count, and `savings_percent`. The percentage is rounded to two decimal places and is `0.0` when the baseline is zero. |
| `warnings` | Fixed, non-sensitive scope, quality, safety, and limit warnings. |
The sampling byte fields use opened-handle snapshot lengths, not physical I/O
counts. The accounting equations are:
```text
discovered_files = selected_files + sampled_out_files
discovered_bytes = planned_bytes + sampled_out_bytes
planned_bytes = scanned_bytes + unscanned_selected_bytes
```
`discovered_bytes` is the sum for eligible files before sampling;
`planned_bytes` is the sum for whole files selected by the file and byte limits;
and `sampled_out_bytes` is the eligible sum omitted by those limits. A selected
file contributes its snapshot length to `scanned_bytes` only when its handle
passes identity/length checks before and after scanning. An unreadable or
unstable selected file contributes that same expected snapshot length to
`unscanned_selected_bytes`, even if a bounded scan read some bytes before
failing. A stable scan can still set `files.limit_hit` when the JSONL line cap
stops parsing; its full snapshot length remains part of the planned/scanned
accounting.
The codecs are independent: JSON minification removes only lexical JSON
whitespace and preserves JSON values but not original bytes; line RLE repeats
only adjacent identical complete line chunks; and template folding stores one
conservative prefix/suffix span for an adjacent run. RLE and template
candidates are decoded locally and must reconstruct the exact input. JSON
candidates are validated as complete JSON, not against the original bytes.
Only a unique candidate with fewer full-item BPE tokens than baseline is
selected. Candidates never stack, and ties fall back to baseline.
JSON minification has a lexical/value-preserving guarantee, not an original-byte
round-trip guarantee: it removes only JSON whitespace outside strings and keeps
string escapes, duplicate keys, key order, and numeric spelling. The baseline
and every candidate are counted as a Codex Responses-style
`function_call_output` item containing `type`, `call_id`, and `output`; stored
tool names and success flags validate the source but do not change that count.
The RLE and template envelopes include fixed instruction strings. Those strings
are part of the candidate output and therefore part of its token count. Their
decoders require the exact marker and instruction, canonical JSON, known fields,
bounded item expansion, and exact reconstruction before a candidate is eligible.
The template codec is a conservative local heuristic, not evidence that a live
provider will natively decode the envelope. Sampling is bounded and deterministic:
newest eligible opened handles are retained within the file cap, files larger
than `--max-bytes` are omitted before they consume a selected slot, and the
remaining candidates are planned in newest order without exceeding the byte cap.
No selected file is split. The resulting report is a sampled, unweighted
experiment rather than a billing or request-volume estimate.
The source JSONL is never copied into the report or another file. It is opened
read-only, scanned within line/file/result bounds, and discarded after counting.
## Related docs
- [Configuration](configuration.md)
- [Tools and safety model](tools-and-safety.md)
- [Sessions, context, and cache](sessions-context-cache.md)
---
[Back to feature docs](README.md) · [Back to repository README](../../README.md)