etoon

Fast TOON (Token-Oriented Object Notation) encoder for Python, Rust, and CLI.

8× faster than toons, 2.7× faster than the official TS SDK, byte-identical output.

中文說明

Performance

Measured on a 50-doc "neptune" payload (7480 bytes JSON → 4012 bytes TOON):

CLI via stdin pipe (Claude / Bash workflows):

Install

Python

pip install etoon

Rust library

cargo add etoon --no-default-features

CLI binary

Download from GitHub Releases, or:

cargo install etoon

Usage

Python

import etoon
docs = [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
print(etoon.dumps(docs))
# [2]{id,name}:
#   1,Alice
#   2,Bob

CLI (Bash pipe)

curl -s https://api.example.com/data | etoon
cat data.json | etoon -o output.toon

LLM agents (Claude Code, etc.)

Add | etoon to any JSON-emitting shell command to save LLM tokens when reading the output. The transformation is lossless and the TOON format is far more compact (40-60% fewer tokens typical).

gh pr list --json number,title,author | etoon
aws s3api list-buckets | etoon
kubectl get pods -o json | etoon

Pipeline rules for LLM use:

• Terminal-only — jq / grep / awk must come before | etoon, never after
• Non-JSON input passes through unchanged (safe to default-on)
• Skip for byte-exact diff comparisons or tiny single-blob outputs
• If etoon fails (rare, on malformed JSON), re-run without the pipe

Example Claude Code rule in CLAUDE.md:

Append | etoon to any JSON-emitting bash command by default. Place it at the very end of the pipeline. Re-run without pipe if it fails.

Rust

let json_bytes = serde_json::to_vec(&my_data)?;
let toon = etoon::toon::encode(&json_bytes)?;

Architecture

Python dict → orjson.dumps → JSON bytes → sonic-rs (SIMD parse) → walk → TOON string

Key optimizations:

• sonic-rs SIMD JSON parser (~7× faster than serde_json)
• orjson bridge — single boundary crossing (vs PyO3-based alternatives)
• uniform-order table fast path — skips 300 key lookups per 50-row table
• itoa specialized integer formatting

Compatibility

Output is byte-identical to the toons Python package (Apache 2.0) and the official toon-format/toon TypeScript SDK. Passes 111/111 TOON spec fixtures covering primitives, objects, arrays (primitive/tabular/nested/bulleted), and whitespace.

Advanced options

# Custom delimiter (saves tokens when values contain commas)
etoon.dumps(data, delimiter="|")   # or "\t"

# Key folding: collapse {a:{b:{c:1}}} → "a.b.c: 1"
etoon.dumps(data, fold_keys=True)
etoon.dumps(data, fold_keys=True, flatten_depth=2)  # partial fold

Limitations

• Integers > 2⁶³ are lossily coerced via f64 (works for most common big integers that happen to be representable; arbitrary-precision is not supported).
• Custom indent is hardcoded to 2 spaces (TOON spec default).

License

Apache 2.0. Test fixtures in tests/fixtures/ are sourced from the toons project (Apache 2.0).