# Tool output compression
[Feature docs index](README.md) · [Repository README](../../README.md)
## Purpose
Reduce provider-visible tokens for high-volume tool results with deterministic summaries while preserving raw local/session tool output.
## Enable
Disabled by default. Opt in through `~/.magi-code/settings.json`:
```json
{
"capabilities": {
"tools": {
"output_compression": { "enabled": true }
}
}
}
```
No slash command, shell proxy, RTK dependency, or per-rule setting controls production compression.
## Scope
| Surface | Behavior |
| --- | --- |
| Provider continuation payload | Curated summary for recognized `bash` commands and the dispatch alias `shell` when enabled. |
| Local terminal/TUI output | Original raw `ToolResult` remains displayed through existing output paths. |
| JSONL sessions | Original raw `ToolResult` is recorded before provider compression. |
| Hooks | Hook payloads and diagnostics use normal raw tool result behavior; compression does not change hook policy. |
| Other tools | Pass through unchanged by compression. A separate failed empty `grep`/`ffgrep` diagnostic path may expose existing stderr/stdout text; it is not compression. |
| Unrecognized commands and unsupported output forms | Pass through unchanged. |
| Compound shell commands | Pass through unchanged for any literal pipeline (`|`) or unsafe operators (`&&`, `||`, `;`, background `&`, file redirects, `<`, `$()`, backticks, newline, carriage return). Exact fd redirects `2>&1` / `1>&2` remain supported; they are the only supported compound shell form. |
| Shell quoting, escaping, expansion, and glob syntax | Pass through unchanged. Single/double quotes, backslash escapes, parameter/command/brace/tilde expansion, glob metacharacters (`*`, `?`, `[`, `]`), and other uncertain shell syntax are never token-classified. |
Compression changes what the next provider request sees. It does not mutate tool execution result, success flag, local transcript, or session artifact already storing/displaying raw output.
Compression is selected only when the complete candidate (including its header) has
strictly fewer locally counted text tokens than the original `ToolResult.content`
under both embedded tokenizers, `o200k_base` and `cl100k_base`. Ties and larger
candidates pass through unchanged. The active model is not available at this
boundary; these are conservative local comparisons, not provider billing guarantees.
Byte counts in the header describe captured streams, not token savings.
Both captured streams must be available as strings in metadata or in complete
`stdout:\n...\nstderr:\n...` content framing. Missing or incomplete framing passes
through unchanged rather than inventing empty streams. Failed results with both
streams empty also pass through, preserving content-only dispatch diagnostics.
The failed empty `grep`/`ffgrep` diagnostic normalization above is an older provider-diagnostic exception: when a search fails with empty `ToolResult.content`, its captured non-empty stderr (or stdout) can be returned so the provider sees the error. This path is separate from curated compression and remains unchanged.
## Offline measurement prototype
The exploratory measurement prototype is run explicitly with:
```text
magi-code sessions measure-compression --model <MODEL>
```
It reads only active primary JSONL files directly under `sessions/` and active
subagent JSONL files directly under `sessions/subagents/`. Discovery traverses
the absolute session path component by component from a stable filesystem
anchor without following ancestor or root symlinks, validates each held
directory handle, opens `subagents/` without following a symlink, enumerates
both directories through held directory capabilities, and opens each candidate
read-only relative to the matching directory capability. Selected files are
not reopened through ambient path access.
Only active-name regular files with stable identity metadata and no more than
the internal 64 MiB per-file cap become eligible. Normal archives, sidecars,
temporary files, history directories, and other non-candidates are ignored
without skipped-input counts. Symlinked active JSONL candidates, unsafe active
file types, and an unsafe `subagents/` path are rejected and counted as
skipped. The command emits one aggregate JSON report with counts and local
token estimates; it does not emit session paths, IDs, raw output, excerpts, or
file contents. The report is privacy-oriented and read-only.
Only supported `openai-codex` text model families are accepted; embedding
models are not. The `o200k_base` family accepts exact `gpt-5`, `gpt-4.1`,
`gpt-4o`, `gpt-4.5`, `o1`, `o3`, and `o4` families with either the exact
family name or a valid `-` suffix; `gpt-5` also accepts version suffixes that
begin with `.`. `codex-` models require a nonempty valid identifier after the
prefix. The `cl100k_base` family accepts exact `gpt-4` and `gpt-3.5-turbo`
families with the same valid `-` suffix rule. Known non-text modality
identifiers (`embedding`, `audio`, `realtime`, `transcribe`, `tts`, and
`image`) are rejected. These are local estimates, not provider billing
counts. An unsupported model is rejected without scanning.
Optional `--max-files` and `--max-bytes` flags bound sampling. `--max-files`
defaults to 100 and has a maximum of 10,000. `--max-bytes` is a byte count;
it defaults to 128 MiB and has a maximum of 1 GiB (with a 1 MiB minimum).
This prototype does not change provider-visible output, production compression,
sessions, settings, authentication, or files under `MC_HOME`. It does not use
the network, does not enable any production codec, and does not implement TOON.
### Measurement report
The JSON report is aggregate-only. Its fields are:
| Field | Meaning |
| --- | --- |
| `prototype` | Always `offline_compression_measurement_v0`; identifies this exploratory report, not a production codec or setting. |
| `schema_version`, `model`, `encoding_family`, `scope`, `exploratory` | Report version, requested model, selected local BPE family, fixed active-session scope, and the fixed exploratory marker. |
| `measurement_unit` | Always `unique_stored_tool_results`; measurements are reported per stored tool result rather than per provider request. |
| `request_weighted` | Always `false`; results are not weighted by request volume. |
| `replay_compaction_applied` | Always `false`; no replay compaction is applied before measurement. |
| `sampling` | File and byte sampling totals. Eligible files are opened and measured from held read-only handles; rejected unsafe files and files over the internal 64 MiB per-file cap are not part of these totals. |
| `files.considered` | Eligible active regular JSONL candidates after safe opening and the per-file cap, before sampling. `scanned` is the number selected and whose handles passed the scan checks; `sampled_out` covers eligible files omitted by the file/byte limits. `skipped` covers rejected active candidates such as symlinks; `unreadable`, `unstable`, and `limit_hit` are separate safety/limit counts. `limit_hit` includes file-level input, result, candidate, and per-file bounds. |
| `jsonl.lines`, `valid_events`, `malformed`, `oversized`, `limit_hit` | Bounded JSONL scan counts. `oversized` counts lines over the line cap; `limit_hit` counts files that reached the line-count cap. |
| `tool_results.found`, `empty`, `measured`, `legacy`, `invalid`, `skipped_limit` | Tool-result counts. Empty results are found and measured when their complete item framing is available, but normally have no codec candidate. `invalid` counts found tool-result events that fail required-field or framing validation; `skipped_limit` counts only known tool results over the candidate-size cap. |
| `baseline` | Raw content bytes and Codex Responses-style local BPE projection tokens for each `function_call_output` item before a codec; these are not full provider-context or billing-token counts. |
| `codecs` | Per-codec attempts, validated candidates, strict improvements, selections, candidate token totals, and selected savings. |
| `portfolio` | Tokens selected after independent per-result selection, saved tokens, transformed output count, and `savings_percent`. The percentage is rounded to two decimal places and is `0.0` when the baseline is zero. |
| `warnings` | Fixed, non-sensitive scope, quality, safety, and limit warnings. |
The sampling byte fields use opened-handle snapshot lengths, not physical I/O
counts. The accounting equations are:
```text
discovered_files = selected_files + sampled_out_files
discovered_bytes = planned_bytes + sampled_out_bytes
planned_bytes = scanned_bytes + unscanned_selected_bytes
```
`discovered_bytes` is the sum for eligible files before sampling;
`planned_bytes` is the sum for whole files selected by the file and byte limits;
and `sampled_out_bytes` is the eligible sum omitted by those limits. A selected
file contributes its snapshot length to `scanned_bytes` only when its handle
passes identity/length checks before and after scanning. An unreadable or
unstable selected file contributes that same expected snapshot length to
`unscanned_selected_bytes`, even if a bounded scan read some bytes before
failing. A stable scan can still set `files.limit_hit` when the JSONL line cap
stops parsing; its full snapshot length remains part of the planned/scanned
accounting.
The codecs are independent: JSON minification removes only lexical JSON
whitespace and preserves JSON values but not original bytes; line RLE repeats
only adjacent identical complete line chunks; and template folding stores one
conservative prefix/suffix span for an adjacent run. RLE and template
candidates are decoded locally and must reconstruct the exact input. JSON
candidates are validated as complete JSON, not against the original bytes.
Only a unique candidate with fewer full-item BPE tokens than baseline is
selected. Candidates never stack, and ties fall back to baseline.
JSON minification has a lexical/value-preserving guarantee, not an original-byte
round-trip guarantee: it removes only JSON whitespace outside strings and keeps
string escapes, duplicate keys, key order, and numeric spelling. The baseline
and every candidate are counted as a Codex Responses-style
`function_call_output` item containing `type`, `call_id`, and `output`; stored
tool names and success flags validate the source but do not change that count.
The RLE and template envelopes include fixed instruction strings. Those strings
are part of the candidate output and therefore part of its token count. Their
decoders require the exact marker and instruction, canonical JSON, known fields,
bounded item expansion, and exact reconstruction before a candidate is eligible.
The template codec is a conservative local heuristic, not evidence that a live
provider will natively decode the envelope. Sampling is bounded and deterministic:
newest eligible opened handles are retained within the file cap, files larger
than `--max-bytes` are omitted before they consume a selected slot, and the
remaining candidates are planned in newest order without exceeding the byte cap.
No selected file is split. The resulting report is a sampled, unweighted
experiment rather than a billing or request-volume estimate.
The source JSONL is never copied into the report or another file. It is opened
read-only, scanned within line/file/result bounds, and discarded after counting.
## Supported first-slice rules
| Rule id | Recognized commands | Summary content |
| --- | --- | --- |
| `bash.git_status` | `git status` or `git --no-pager status` with `--short`, `--porcelain`, `--porcelain=v1`, or `-sb`; `--branch` may accompany one of these forms | Changed-file count, untracked count, branch/ahead-behind line when present, first capped status lines. Bare long-form status and `--branch` alone pass through. |
| `bash.git_diff` | `git diff ...`, `git --no-pager diff ...`, or exactly one of `git -C <path> diff ...`, `git -C <path> --no-pager diff ...`, and `git --no-pager -C <path> diff ...`, with one unquoted cwd token using only ASCII letters, digits, `/`, `.`, `_`, or `-` and not starting with `-`, plus supported patch-shaped output | Files touched from `diff --git`, hunk count, insertion/deletion counts, first capped file list, capped stat lines when present. Empty diffs pass through because the summary header costs more tokens. |
| `bash.git_log` | `git log ... --oneline ...` or `git --no-pager log ... --oneline ...` without patch/stat/graph, decoration/parent/child, or output-format overrides | Non-empty commit line count and first 15 oneline subjects. Every non-empty output line must have an oneline commit-hash shape; otherwise the raw result passes through. |
| `bash.cargo_check` | Human-readable `cargo check ...` or strict `cargo +<toolchain> check ...` without machine-readable or informational/output-shaping flags | Exit code, warning/error counts, compiler error codes, final cargo status, first capped diagnostics. The toolchain token is one `+` plus only ASCII letters, digits, `.`, `_`, or `-`. |
| `bash.cargo_test` | Human-readable `cargo test ...` or strict `cargo +<toolchain> test ...` without machine-readable output flags, libtest `--list`/`--format` output-shaping flags (split or equal forms, including after `--`), or `--nocapture`/`--show-output`/`--no-capture` | Exit code, warning/error counts, compiler error codes, `test result:` lines, failing test names, first capped failure/panic context. |
`bash` and `shell` use the same curated rule ids. The provider header reports the actual dispatch name (`bash` or `shell`).
Cargo `-h`, `--help`, `--timings` (including equal forms), `--future-incompat-report`, and `--color=always` / split `--color always` pass through unchanged for both check and test, including after `--`. Existing machine formats and the test output controls above also pass through unchanged. Normal `--color=auto` and `--color=never` remain compressible.
Git cwd qualification accepts only the three prefix orders in the table. The path must be one unquoted non-empty token using only ASCII letters, digits, `/`, `.`, `_`, or `-`, and must not begin with `-`; quoted, spaced, escaped, expanded, globbed, attached-`-C`, repeated-`-C`, or otherwise uncertain forms pass through. Git diff option checks stop at the first exact `--`: incompatible output and global options before it pass through, while tokens after it are pathspecs even when they look like `--name-only`, `--no-pager`, or `-C...`. No cwd-qualified status or log rule is added.
Recognition uses conservative whitespace tokenization plus a shell-syntax preflight. Commands containing quotes, backslash escapes, parameter/command/brace/tilde expansion, glob syntax, or other uncertain shell syntax pass through unchanged. Exact `2>&1` and `1>&2` fd redirect tokens are ignored for base command classification. Any literal pipe (`|`) passes through unchanged. If classification or output validation is uncertain, magi-code sends original output to provider.
All Git rules pass through the exact raw `result.content` whenever captured stderr is non-empty, preserving Git errors and warnings. A nonzero exit code alone does not disable `bash.git_diff`: a validated patch with empty stderr, such as `git diff --exit-code` or a supported cwd-qualified equivalent, remains compressible.
## Provider-visible format
When compression applies, provider receives deterministic text:
```text
[tool_output_compression]
tool: bash
rule: bash.cargo_check
command: cargo check --all-targets
success: false
exit_code: 101
raw_stdout_bytes: 0
raw_stderr_bytes: 34812
stdout_truncated: false
stderr_truncated: false
compression: curated
exit_code: 101
warnings: 1
errors: 2
compiler_error_codes: E0425
final_status: error: could not compile `demo`
diagnostics:
- error[E0425]: cannot find value `x` in this scope
- warning: unused import: `Foo`
- error: could not compile `demo`
```
Header fields preserve operational context:
| Field | Meaning |
| --- | --- |
| `tool` | Actual dispatch name; `bash` or its `shell` alias when compression applies. |
| `rule` | Curated rule id used to summarize output. |
| `command` | Original trimmed command text from tool call arguments. |
| `success` | Original tool success flag. |
| `exit_code` | Bash process exit code, or `null` when unavailable. |
| `raw_stdout_bytes` / `raw_stderr_bytes` | Byte counts from raw captured streams. |
| `stdout_truncated` / `stderr_truncated` | Raw stream truncation flags from bash metadata. |
| `compression` | Constant marker: `curated`. |
## Examples
### `git status --short --branch`
Raw stdout:
```text
## main...origin/main [ahead 1]
M src/lib.rs
?? notes.md
```
This short result passes through unchanged: adding the compression header would
cost more tokens. Larger status results can produce a summary containing file and
untracked counts, the branch line, and the first capped status lines.
### `git diff -- src/lib.rs`
Summary counts files, hunks, insertions, deletions, and first capped files/stat lines. It does not attempt semantic diff interpretation.
`git --no-pager diff` is recognized the same way. A strict cwd-qualified form such as `git -C ./repo --no-pager diff -- src/lib.rs` is also recognized; `git --no-pager -C ./repo diff` uses the other accepted global-option order. Git diff validation stops at the first exact `--`, so option-looking filenames after it do not disable patch compression. Any pipeline such as `git diff | cat` passes through unchanged, regardless of its filter, because the classifier does not attempt to prove that a pipeline preserves output shape.
### `git log --oneline -20`
Provider-visible summary counts non-empty oneline commit records and includes the first 15 entries under `subjects:`. Non-oneline forms such as `git log -5`, `git log --pretty=oneline`, `git log --oneline --stat`, `git log --oneline --decorate=full`, and parent/child forms pass through unchanged.
### Passthrough example: `cargo +nightly-2025-01-01 test -- --no-capture`
This command is a passthrough example, not a compressed example. Strictly valid toolchain-qualified check/test commands use the same human-readable summaries as their unqualified forms. Help, timing, future-incompatibility, always-color, machine-format, list/format, and test-output-control forms pass through exactly so their output shape is preserved. A command such as `cargo +stable build`, a malformed `+toolchain`, or a qualified command with uncertain syntax also passes through.
### `cargo test tool_output_compression`
Summary keeps failing test names and first panic/failure lines so provider can continue debugging without receiving entire test logs.
## Developer extension process
Implementation lives at the provider-visible boundary:
| File | Role |
| --- | --- |
| `src/config/settings.rs` | Defines the `ToolOutputCompressionSettings` setting and its default-disabled serde/schema behavior. |
| `src/tools/runtime.rs` | Stores the setting in `ToolRuntime` and exposes the crate-private accessor used by the lifecycle boundary. |
| `src/agent/tool_lifecycle.rs` | Records raw `ToolResult` to session, then builds provider output with compression setting before `ProviderToolResult`. |
| `src/agent/tool_output_compression.rs` | Pure classifier and summarizers for curated bash/shell rules. |
| `config/example.settings.json` | Shows default-disabled setting. |
To add rule:
1. Add conservative classifier branch in `classify_bash_command`.
2. Reject compound shell operators unless rule explicitly handles them safely.
3. Summarize only existing `ToolResult.metadata` / `ToolResult.content`; never re-run command.
4. Keep output deterministic: stable field order, fixed caps, no timestamps unless already in raw output and useful.
5. Add unit tests for disabled passthrough, non-match passthrough, matched summary, missing-stream diagnostics, short-result passthrough, and compound-command passthrough. Keep the full-header token savings gate for every rule.
6. Update this doc and [Tools and safety model](tools-and-safety.md) if provider-visible behavior changes.
## Limitations
- No arbitrary shell-output compression.
- No curated compression for `read`, `grep`, `hash_edit`, `write`, `subagents`, or `web`. Failed empty `grep`/`ffgrep` results still have the separate diagnostic normalization described above.
- No provider billing-token guarantee. Selection requires strict text-token savings under both embedded tokenizers, not the active provider's full request accounting. Other model tokenizers may differ.
- No shell parsing beyond conservative token/operator checks. Exact fd redirects `2>&1` / `1>&2` are the only supported compound shell forms; commands containing quotes, escapes, parameter/command/brace/tilde expansions, glob syntax, or other uncertain shell syntax pass through unchanged, and every literal pipeline passes through unchanged.
- Git rules are intentionally narrow: status requires an explicit short/porcelain form; diff requires supported patch-shaped output and passes through output-formatting/indicator and global options before the first exact `--`, while tokens after `--` are treated as pathspecs; cwd-qualified diff accepts only the three strict `-C <path>` prefix orders with one conservative path token; log requires plain oneline output with no patch/stat/graph, decoration, parent/child, or format override. Qualified status/log forms are not recognized.
- Cargo machine-readable message formats, `-h`/`--help`, `--timings` and equal forms, `--future-incompat-report`, `--color=always` and split `--color always`, libtest `--list`, every libtest `--format` form (split/equal, including after `--`), and test output controls (`--nocapture`, `--show-output`, `--no-capture`, including after `--`) pass through unchanged. Strict toolchain qualification is limited to `cargo +<toolchain> check/test` with the documented ASCII token grammar.
- No per-rule settings or user-editable caps.
- Raw output can still be large in local display/session artifacts because preservation is intentional.
## Related docs
- [Configuration](configuration.md)
- [Tools and safety model](tools-and-safety.md)
- [Sessions, context, and cache](sessions-context-cache.md)
---
[Back to feature docs](README.md) · [Back to repository README](../../README.md)