delta-pack 0.2.0

# delta-pack

Binary serialization with delta compression for real-time state synchronization.

Delta-Pack is a cross-language serialization framework optimized for networked applications where you need to efficiently synchronize state between clients and servers. It provides both full encoding and delta encoding (only transmitting what changed).

## Features

- **Compact binary format** - Smaller than JSON, MessagePack, and often Protobuf
- **Delta compression** - Encode only the differences between two states
- **Cross-language** - Compatible with TypeScript and C# implementations
- **Zero-copy strings** - String dictionary deduplication within each message
- **Serde integration** - Generated types derive `Serialize`/`Deserialize`

## Installation

Add to your `Cargo.toml`:

```toml
[dependencies]
delta-pack = "0.1"
```

## Usage

Delta-Pack uses code generation from YAML schemas. Define your schema, generate Rust code, and use the generated types.

### 1. Define a Schema

```yaml
# schema.yml
HairColor:
  - BLACK
  - BROWN
  - BLOND
  - RED

Address:
  street: string
  city: string
  zip: string

User:
  id: string
  name: string
  age: uint
  weight: float(precision=0.01)
  hairColor: HairColor
  address: Address?
  tags: string[]
  metadata: <string, string>
```

### 2. Generate Rust Code

```bash
delta-pack generate schema.yml -l rust > src/generated.rs
```

### 3. Use the Generated Types

```rust,ignore
use crate::generated::{User, Address, HairColor};

fn main() {
    // Create a user
    let user1 = User {
        id: "user-123".into(),
        name: "Alice".into(),
        age: 30,
        weight: 65.5,
        hair_color: HairColor::Brown,
        address: Some(Address {
            street: "123 Main St".into(),
            city: "Springfield".into(),
            zip: "12345".into(),
        }),
        tags: vec!["admin".into(), "verified".into()],
        metadata: [("level".into(), "5".into())].into(),
    };

    // Full encode/decode
    let bytes = user1.encode();
    let decoded = User::decode(&bytes);
    assert!(user1.equals(&decoded));

    // Delta encoding - only send what changed
    let user2 = User {
        age: 31,  // birthday!
        ..user1.clone()
    };

    let full_size = user2.encode().len();
    let diff = User::encode_diff(&user1, &user2);
    let diff_size = diff.len();

    println!("Full: {} bytes, Diff: {} bytes", full_size, diff_size);
    // Full: 58 bytes, Diff: 3 bytes

    // Apply the diff to reconstruct user2
    let reconstructed = User::decode_diff(&user1, &diff);
    assert!(user2.equals(&reconstructed));
}
```

## Generated API

Every generated type provides these methods:

| Method                                       | Description                                 |
| -------------------------------------------- | ------------------------------------------- |
| `encode(&self) -> Vec<u8>`                   | Serialize to binary                         |
| `decode(buf: &[u8]) -> Self`                 | Deserialize from binary                     |
| `encode_diff(a: &Self, b: &Self) -> Vec<u8>` | Encode only the differences from `a` to `b` |
| `decode_diff(a: &Self, diff: &[u8]) -> Self` | Apply a diff to `a` to produce `b`          |
| `equals(&self, other: &Self) -> bool`        | Deep equality (respects float precision)    |

Generated types also derive:

- `Clone`, `Debug` - Standard traits
- `Default` - All fields initialized to zero/empty values
- `Serialize`, `Deserialize` - Serde support for JSON interop

## Schema Types

### Primitives

| Schema                  | Rust Type | Notes                        |
| ----------------------- | --------- | ---------------------------- |
| `string`                | `String`  | UTF-8, dictionary-compressed |
| `int`                   | `i64`     | Signed, varint-encoded       |
| `uint`                  | `u64`     | Unsigned, varint-encoded     |
| `int(min=0, max=100)`   | `u64`     | Bounded, more compact        |
| `float`                 | `f32`     | 32-bit IEEE 754              |
| `float(precision=0.01)` | `f32`     | Quantized for smaller diffs  |
| `boolean`               | `bool`    | 1 bit, RLE-compressed        |

### Containers

| Schema   | Rust Type       |
| -------- | --------------- |
| `T[]`    | `Vec<T>`        |
| `T?`     | `Option<T>`     |
| `<K, V>` | `IndexMap<K, V>` |

### Named Types

```yaml
# Enum (list of strings)
Direction:
  - up
  - down
  - left
  - right

# Object (key-value properties)
Player:
  name: string
  score: uint
  position: Position

# Union (list of type references)
Message:
  - ChatMessage
  - MoveMessage
  - AttackMessage
```

### Self-References

Recursive types are supported and generate `Box<T>`:

```yaml
TreeNode:
  value: int
  children: TreeNode[] # Generates Vec<Box<TreeNode>>
```

## Binary Format

```text
[data section][RLE section][numRleBits: reverse varint]
```

- **Data section**: Primitives encoded sequentially (strings with dictionary, varints, floats)
- **RLE section**: Run-length encoded bits (booleans, optional flags, change indicators)
- **Reverse varint**: Bit count stored at end for streaming decode

## Performance

Benchmarks comparing encode throughput (higher is better):

| Schema     | DeltaPack   | JSON  | MessagePack |
| ---------- | ----------- | ----- | ----------- |
| Primitives | 34.1M ops/s | 11.1M | 10.1M       |
| GameState  | 5.8M ops/s  | 673K  | 816K        |
| User       | 5.3M ops/s  | 1.8M  | 2.0M        |

Run benchmarks:

```bash
cd rust/benchmarks
./build.sh  # Generate benchmark schemas
cargo run --release
```

### Faster Decoding with mimalloc

Decode performance is allocation-bound. Using mimalloc instead of the system allocator improves decode throughput by ~30%:

```toml
# Cargo.toml
[dependencies]
mimalloc = "0.1"
```

```rust,ignore
// main.rs
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
```

## Cross-Language Compatibility

Delta-Pack ensures binary compatibility across Rust, TypeScript, and C#:

- Same schema produces interoperable binary format
- Conformance tests verify encode/decode compatibility
- Diff encoding uses sorted keys for deterministic output

All languages use insertion-order-preserving maps (`IndexMap` in Rust, `Map` in TypeScript, `OrderedDictionary` in C#), producing deterministic encoding for the same insertion order.

## License

MIT