TOON Format - Token-Oriented Object Notation

A Rust implementation of the TOON (Token-Oriented Object Notation) format.

TOON is a compact, human-readable serialization format designed specifically to minimize tokens when sending structured data to Large Language Models (LLMs). It achieves 30-60% token reduction compared to JSON for uniform arrays while maintaining full lossless round-trip capability.

Features

• 🚀 Compact: YAML-like indentation + CSV-like tabular arrays
• 🔄 Lossless: JSON → TOON → JSON without data loss
• 📊 Schema-aware: Explicit array lengths [N] and field lists {fields}
• 🧠 LLM-optimized: Designed specifically for LLM prompts
• ⚡ Fast: Zero-copy parsing with minimal allocations
• 🔧 Flexible: Multiple delimiters (comma, tab, pipe)

Quick Start

use toon_format_rs::{parse, to_string, Value};

// Parse TOON string
let input = r#"id: 123
name: Alice
active: true"#;

let value = parse(input).unwrap();
assert_eq!(value.get("id").unwrap().as_i64(), Some(123));

// Serialize to TOON
let output = to_string(&value).unwrap();
assert!(output.contains("id: 123"));

Examples

Simple Object

TOON:

id: 123
name: Alice
active: true

Equivalent JSON:

{"id": 123, "name": "Alice", "active": true}

Nested Object

TOON:

user:
  id: 1
  name: Alice

Inline Array

TOON:

tags[3]: foo,bar,baz

Equivalent JSON:

{"tags": ["foo", "bar", "baz"]}

Tabular Array (Major Token Savings!)

TOON:

users[2]{id,name,role}:
  1,Alice,admin
  2,Bob,user

Equivalent JSON:

{"users": [
  {"id": 1, "name": "Alice", "role": "admin"},
  {"id": 2, "name": "Bob", "role": "user"}
]}

Complex Document

TOON:

context:
  task: Our favorite hikes together
  location: Boulder
  season: spring_2025
friends[3]: ana,luis,sam
hikes[3]{id,name,distanceKm,elevationGain,companion,wasSunny}:
  1,Blue Lake Trail,7.5,320,ana,true
  2,Ridge Overlook,9.2,540,luis,false
  3,Wildflower Loop,5.1,180,sam,true

JSON Interoperability

use toon_format_rs::{json_to_toon, toon_to_json};

// Convert JSON to TOON
let json = r#"{"id":123,"name":"Alice"}"#;
let toon = json_to_toon(json).unwrap();

// Convert TOON back to JSON
let json_back = toon_to_json(&toon).unwrap();

Installation

Add to your Cargo.toml:

[dependencies]
toon-format-rs = "0.1"

Why TOON?

Benchmark: 100 uniform employee records

Specification

This implementation follows the TOON Specification v3.0.

Key features:

• Indentation-based objects: No braces, YAML-like nesting
• Explicit array headers: key[N]: for inline arrays, key[N]{fields}: for tabular arrays
• Minimal quoting: Strings only quoted when necessary
• Multiple delimiters: Comma ,, tab \t, or pipe |
• Canonical numbers: No scientific notation, no trailing zeros

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Acknowledgments

• TOON Format Organization for the specification
• Inspired by JSON, YAML, and CSV formats