shrinkfit

Shrink a JSON (or YAML) document to fit a character budget while keeping as much of its schema and structure as possible.

Built for AI-agent tools that return large structured payloads. Instead of failing, or spilling the result to a file, a tool can always answer inline with a faithful, clearly marked sample of the data:

$ shrinkfit --max-chars 900 api_response.yaml   # 25 KB in, 895 chars out
status: ok
total: 250
users:
  - id: 0
    name: User 0
    email: user0@example.com
    roles:
      - dev
      - admin
  - id: 1
    name: User 1
    email: user1@example.com
    roles:
      - dev
  - id: 2
    name: User 2
    email: user2@example.com
    roles:
      - dev
  - …[246 items omitted]
  - id: 249
    name: User 249
    email: user249@example.com
    roles:
      - dev
changelog: |-
  ## v1.12
  Fixed a number of bugs and improved performance across the board. Fixed a number of bugs…[+176 chars]

  ## v1.11
  Fixed a…[+257 chars]

  ## v1.10
  Fixed a…[+257 chars]

  … (one heading + preview per remaining release, down to v1.1)

The input was 250 user records plus a 12-release markdown changelog. Every field survives. The first records are kept whole, and so is the last one, so you can see where the list ends. Every changelog heading is still present, with a preview of its section. (If some records had different keys, the first record of each distinct shape would be preferred too.)

• Hard budget. The output is always at most max_chars characters, as long as the budget can hold the document's most compact form. JSON sizes are predicted exactly; for YAML the real output is measured and the budget corrected.
• Always valid. Output parses in the requested format. Cuts are marked in-band with … markers, so a reader never mistakes a sample for the whole: "…[118 items omitted]", "…": "12 keys omitted", …[+2133 chars].
• Schema first. Every key of an object is kept while it can be, scalars survive intact, and array sampling prefers items with new shapes.
• Structure-aware strings. Long text keeps its outline (headings, lines, paragraphs) and shows a preview of every section rather than just a prefix.
• Fast and dependency-free. Rust core with no dependencies; a 41 MB payload shrinks in about 0.5 s. Thin bindings for Python and TypeScript (WASM).

Usage

Python

from shrinkfit import shrink, shrink_with_report

shrink(big_dict)                                 # -> str: indented JSON, 10000-char budget
shrink(big_dict, max_chars=4000, indent=0)       # compact one-line JSON
shrink(big_dict, 4000, format="yaml")            # YAML output
shrink(json_or_yaml_text, 4000)                  # str/bytes: format auto-detected,
                                                 # output in the same format

r = shrink_with_report(big_dict, 4000, string_section_chars=60)
r.text, r.report["items_omitted"], r.report["fits"]

TypeScript / JavaScript (Node, Deno, browsers)

import { init, shrink, shrinkWithReport } from "shrinkfit";

await init();                      // loads shrinkfit.wasm next to the module
shrink(bigObject);                 // 10000-char budget, indented JSON
shrink(bigObject, 4000);
shrink(bigObject, 4000, { format: "yaml", stringSectionChars: 60 });

Rust

use shrinkfit::{shrink_json, Options};

let out = shrink_json(&big_json_text, &Options::new(4000))?; // or Options::default()
println!("{}", out.text);
// YAML text in (feature `yaml`): set `opts.input_format = InputFormat::Yaml`
// and call `shrink_str`.
assert!(out.report.fits);

CLI

cargo install --path crates/shrinkfit-cli
shrinkfit [--max-chars 10000] [--format auto|json|yaml] [--input-format auto|json|yaml] \
          [--indent 2] [--report] [payload.json|-]

The input format is auto-detected (JSON first, then YAML) and the output uses the same format unless --format says otherwise.

Playground

shrinkfit serve [--port 8765]    # then open http://127.0.0.1:8765/

A dependency-free local web page for exploring the behaviour while developing: paste JSON or YAML, change any option, and see the output and report update live. It talks to the same Rust code through the same options object as the bindings. It binds to loopback only; it is a dev tool, not a production server.

How it works

In brief (see docs/ALGORITHM.md for the details):

1. Size every subtree once. Both its full size and its minimum size (the smallest form it could be reduced to) are recorded.
2. Allocate the budget top-down. An object keeps all its keys while its skeleton (every key, each value at its minimum) fits. Spare budget is shared among its values by water-filling: small values are kept whole, and the rest is divided equally among the larger ones, with any unused share flowing on to the others.
3. Sample arrays by priority: the first item, then the first item of each distinct shape, then the last item, then the rest in order. As many whole items as fit are kept, plus one reduced item if budget remains. When not even one item fits whole, a few shape-distinct items are kept in reduced form. Each gap is replaced by a "…[N items omitted]" marker.
4. Truncate strings by structure. Text is split at its coarsest natural boundary (blank lines, then lines, then sentences). Whole sections are shown up to a head of string_head_chars. After that, each section shows its first string_section_chars and a …[+N chars] marker. Both limits scale together to use the budget fully. Only under severe pressure are trailing sections dropped behind …[+K sections, N chars].
5. Verify. The result is rendered and measured. If the format's size model was only an estimate (YAML), the internal budget is searched until the real output fits as tightly as possible.

Options

The report includes format (the output format actually used), input_chars, output_chars, fits, pruned, strings_truncated, arrays_sampled, items_omitted, objects_trimmed, keys_omitted and containers_collapsed.

Repository layout

crates/shrinkfit       core library (Rust API), zero dependencies
crates/shrinkfit-cli   command-line tool
crates/shrinkfit-ffi   C ABI (cdylib/staticlib + include/shrinkfit.h), used by all bindings
bindings/python        ctypes wrapper + wheel build (no PyO3 needed)
bindings/typescript    WASM loader, no wasm-bindgen needed

Every binding calls one function, shrinkfit_shrink(input_json, options_json) -> envelope_json. Options pass through as JSON, so adding an option never requires binding changes.

Status and roadmap

v0.1 is a first working cut. Planned next:

• More input formats: TOML and JSON Lines are natural next steps (YAML input is already native; see input_format).
• Token budgets: a pluggable cost function (e.g. a tokenizer) instead of characters.
• Optionally listing omitted key names in the object marker.
• Published wheels (manylinux/macOS/Windows via cibuildwheel), crate and npm package.

License

Apache-2.0. See LICENSE.