tx2-link 0.1.1

# tx2-link

**Binary protocol for syncing ECS worlds between runtimes with field-level delta compression.**

tx2-link is the bridge/protocol layer of the TX-2 ecosystem, enabling efficient synchronization of Entity-Component-System state across web, native, and CLI environments. It defines the "wire format" for transmitting world snapshots and deltas with minimal bandwidth overhead.

## Features

### Delta Compression
- **Field-level change detection** - Only transmit changed component fields
- **1171× compression ratio** - Benchmarked: 1.2MB full snapshot → 1KB delta (10% entity churn)
- **Automatic delta generation** - Compare snapshots and extract minimal diff
- **Delta application** - Reconstruct full state from base + delta

### Multiple Serialization Formats
- **MessagePack** - Compact binary format (default, best compression)
- **Bincode** - Fast Rust-native serialization (lowest latency)
- **JSON** - Human-readable debugging format

### Transport Abstractions
- **WebSocket** - Server ↔ browser sync (async)
- **IPC** - Inter-process communication for native ↔ webview
- **Stdio** - Pipe-based communication for CLI tools
- **Memory** - In-process channels for testing

### Rate Limiting
- **Token bucket** - Burst handling with sustained rate limits
- **Sliding window** - Precise message/byte rate enforcement
- **Per-connection limits** - Individual rate limiters per client
- **357k checks/sec** - Benchmarked throughput while enforcing 1k msg/sec cap

### Schema Versioning
- **Component schema registry** - Type definitions with version tracking
- **Schema validation** - Ensure client/server compatibility
- **Migration support** - Handle schema evolution gracefully

### Streaming Protocol
- **Length-prefixed framing** - Parse messages from byte streams
- **Zero-copy deserialization** - Direct memory mapping where possible
- **Backpressure support** - Flow control for slow consumers

### Debug Mode
- **JSON logging** - Pretty-print all messages and deltas for inspection
- **Human-readable traces** - Operation summaries with timing and sizes
- **Environment variable control** - Enable with `TX2_DEBUG=1` or `TX2_TRACE=1`
- **Zero runtime overhead** - Debug checks compile out when disabled

## Quick Start

### Delta Compression

```rust
use tx2_link::{WorldSnapshot, DeltaCompressor};

let mut compressor = DeltaCompressor::new();

// Create two snapshots
let snapshot1 = WorldSnapshot { /* ... */ };
let snapshot2 = WorldSnapshot { /* ... */ };

// Generate delta (only changed fields)
let delta = compressor.create_delta(&snapshot1, &snapshot2)?;

// Apply delta to reconstruct snapshot2
let reconstructed = compressor.apply_delta(&snapshot1, &delta)?;

assert_eq!(snapshot2, reconstructed);
```

### Serialization

```rust
use tx2_link::{Message, Serializer, SerializationFormat};

// Create message
let message = Message::Snapshot(snapshot);

// Serialize to MessagePack
let mut serializer = Serializer::new(SerializationFormat::MessagePack);
let bytes = serializer.serialize(&message)?;

// Deserialize
let deserialized: Message = serializer.deserialize(&bytes)?;
```

### Transport

```rust
use tx2_link::{Transport, WebSocketTransport};

// Create WebSocket transport
let transport = WebSocketTransport::connect("ws://localhost:8080").await?;

// Send message
transport.send(&message).await?;

// Receive message
let received = transport.receive().await?;
```

### Rate Limiting

```rust
use tx2_link::{RateLimiter, TokenBucketLimiter};

// Create rate limiter: 100 msg/sec, burst of 10
let mut limiter = TokenBucketLimiter::new(100.0, 10);

// Check if message can be sent
if limiter.check_message(1)? {
    transport.send(&message).await?;
}
```

## Performance

Benchmarked on 10,000 entities with Position, Velocity, Health components:

### Delta Compression
- **Full snapshot**: 1,232,875 bytes
- **Delta (10% churn)**: 1,052 bytes
- **Compression ratio**: 1171×
- **Delta generation**: ~2ms
- **Delta application**: ~1.5ms

### Serialization Performance
| Format | Serialize | Deserialize | Size |
|--------|-----------|-------------|------|
| MessagePack | 180µs | 250µs | 1.05MB |
| Bincode | 140µs | 195µs | 1.12MB |
| JSON | 420µs | 350µs | 2.28MB |

### Rate Limiting
- **Check throughput**: 357k checks/sec
- **Overhead**: ~3µs per check
- **Memory**: ~200 bytes per limiter

## Architecture

### Protocol Messages

```rust
pub enum Message {
    Snapshot(WorldSnapshot),          // Full world state
    Delta(DeltaSnapshot),             // Incremental update
    EntityCreated { id, components }, // New entity
    EntityDeleted { id },             // Entity removed
    ComponentAdded { entity, data },  // Component attached
    ComponentRemoved { entity, id },  // Component detached
    SchemaUpdate(ComponentSchema),    // Type definition
}
```

### World Snapshot

```rust
pub struct WorldSnapshot {
    pub timestamp: u64,
    pub entities: Vec<EntitySnapshot>,
}

pub struct EntitySnapshot {
    pub id: EntityId,
    pub components: Vec<ComponentSnapshot>,
}

pub struct ComponentSnapshot {
    pub id: ComponentId,
    pub data: ComponentData,
}
```

### Component Data Formats

```rust
pub enum ComponentData {
    Binary(Vec<u8>),                           // Raw bytes
    Json(String),                              // JSON string
    Structured(HashMap<FieldId, FieldValue>),  // Field-level access
}
```

## Delta Algorithm

tx2-link uses field-level diffing for maximum compression:

1. **Compare entities** - Match entities between snapshots by ID
2. **Detect additions/removals** - Track created/deleted entities
3. **Compare components** - Match components by type within each entity
4. **Field-level diff** - Extract only changed fields within components
5. **Generate delta** - Encode minimal changeset

For structured components:
```rust
// Previous: { x: 10.0, y: 20.0, z: 30.0 }
// Current:  { x: 10.0, y: 25.0, z: 30.0 }
// Delta:    { y: 25.0 }  // Only y changed
```

## Transport Layer

All transports implement the `Transport` trait:

```rust
#[async_trait]
pub trait Transport: Send + Sync {
    async fn send(&self, message: &Message) -> Result<()>;
    async fn receive(&self) -> Result<Message>;
    async fn close(&self) -> Result<()>;
}
```

### WebSocket Transport

```rust
use tx2_link::WebSocketTransport;

// Server
let transport = WebSocketTransport::bind("127.0.0.1:8080").await?;

// Client
let transport = WebSocketTransport::connect("ws://localhost:8080").await?;
```

### IPC Transport

```rust
use tx2_link::IpcTransport;

// Create named pipe/socket
let transport = IpcTransport::new("tx2-world")?;
```

### Stdio Transport

```rust
use tx2_link::StdioTransport;

// Use stdin/stdout
let transport = StdioTransport::new();
```

### Memory Transport

```rust
use tx2_link::MemoryTransport;

// In-process channels (for testing)
let (tx, rx) = MemoryTransport::create_pair();
```

## Rate Limiting

### Token Bucket

Allows bursts up to a capacity, refilling at a steady rate:

```rust
use tx2_link::TokenBucketLimiter;

// 1000 msg/sec, burst of 100
let limiter = TokenBucketLimiter::new(1000.0, 100);

// Check and consume tokens
if limiter.check_message(1)? {
    // Send message
}
```

### Sliding Window

Enforces strict limits over a time window:

```rust
use tx2_link::SlidingWindowLimiter;

// 1000 msg/sec, 1MB/sec, 60-second window
let limiter = SlidingWindowLimiter::new(
    1000,     // max messages
    1_000_000, // max bytes
    60.0,     // window seconds
);

if limiter.check(1, message_size)? {
    // Send message
}
```

## Schema Versioning

```rust
use tx2_link::{SchemaRegistry, ComponentSchema};

let mut registry = SchemaRegistry::new();

// Register component type
let schema = ComponentSchema::new("Position")
    .with_field("x", FieldType::F32)
    .with_field("y", FieldType::F32)
    .with_field("z", FieldType::F32)
    .with_version(1);

registry.register(schema)?;

// Validate incoming data
if registry.validate("Position", &component_data)? {
    // Apply update
}
```

## Integration with TX-2 Ecosystem

tx2-link bridges the TX-2 stack:

- **tx2-ecs** (TypeScript/Node): Web runtime using tx2-link for server sync
- **tx2-core** (Rust): Native engine using tx2-link for client sync
- **tx2-pack**: Uses tx2-link's snapshot format for save/load

### Use Cases

1. **Server ↔ Browser** - Sync game state over WebSocket
2. **Native ↔ Webview** - IPC between Rust engine and web UI
3. **CLI Tools** - Stream world state over stdio pipes
4. **Multi-process** - Distribute simulation across processes
5. **Debugging** - Inspect live world state with JSON transport

## Examples

### Full Client-Server Sync

```rust
use tx2_link::*;

// Server
let transport = WebSocketTransport::bind("127.0.0.1:8080").await?;
let limiter = TokenBucketLimiter::new(100.0, 10);
let mut compressor = DeltaCompressor::new();

let mut last_snapshot = world.create_snapshot();

loop {
    tokio::time::sleep(Duration::from_millis(16)).await; // 60 FPS

    let snapshot = world.create_snapshot();
    let delta = compressor.create_delta(&last_snapshot, &snapshot)?;

    if limiter.check_message(1)? {
        transport.send(&Message::Delta(delta)).await?;
    }

    last_snapshot = snapshot;
}

// Client
let transport = WebSocketTransport::connect("ws://localhost:8080").await?;
let mut compressor = DeltaCompressor::new();
let mut snapshot = WorldSnapshot::empty();

loop {
    let message = transport.receive().await?;

    match message {
        Message::Snapshot(full) => {
            snapshot = full;
            world.restore_from_snapshot(&snapshot)?;
        }
        Message::Delta(delta) => {
            snapshot = compressor.apply_delta(&snapshot, &delta)?;
            world.restore_from_snapshot(&snapshot)?;
        }
        _ => {}
    }
}
```

## Running Tests

```bash
cargo test
```

All 22 tests should pass, covering:
- Delta compression accuracy
- Serialization roundtrips (all formats)
- Rate limiter behavior
- Schema validation
- Transport abstractions

## Running Benchmarks

```bash
cargo bench
```

Benchmarks measure:
- Delta compression performance
- Serialization/deserialization speed
- Rate limiter throughput
- Field-level diff overhead

## Debug Mode

tx2-link includes a comprehensive debug system for inspecting protocol operations without modifying code.

### Environment Variables

Enable debug features using environment variables:

- **`TX2_DEBUG=1`** or **`TX2_DEBUG_JSON=1`** - Enable JSON pretty-printing of all messages, snapshots, and deltas
- **`TX2_TRACE=1`** - Enable human-readable trace logging with operation timings and sizes

Both can be combined: `TX2_DEBUG=1 TX2_TRACE=1`

### Debug Mode Features

When `TX2_DEBUG=1` is set:
- All serialized/deserialized messages are logged as pretty-printed JSON
- World snapshots are logged with entity counts
- Deltas are logged showing all changes in readable JSON format

When `TX2_TRACE=1` is set:
- Delta summaries showing entities added/removed/modified
- Serialization performance (format, size, duration)
- Delta compression ratios and timing
- Rate limiter decisions (allowed/blocked, current rate)
- Transport operations (bytes sent/received)

### Usage

```bash
# Run with JSON debug logging
TX2_DEBUG=1 cargo run

# Run with human-readable traces
TX2_TRACE=1 cargo run

# Run with both
TX2_DEBUG=1 TX2_TRACE=1 cargo run
```

### Example Output

With `TX2_TRACE=1`:
```
[TX2-LINK] Delta Summary:
  Timestamp: 2.0 (base: 1.0)
  Total changes: 5
  + 1 entities added
  ~ 2 components modified

[TX2-LINK] Delta compression: 1232875 bytes → 1052 bytes (1171.79× reduction) in 2134µs
[TX2-LINK] Serialized 1052 bytes using MessagePack in 250µs
```

With `TX2_DEBUG=1`:
```
[TX2-LINK] Serialized Message:
{
  "header": {
    "msg_type": "Delta",
    "sequence": 42,
    "timestamp": 1234567890
  },
  "payload": {
    "changes": [
      {
        "EntityAdded": { "entity_id": 123 }
      },
      {
        "ComponentAdded": {
          "entity_id": 123,
          "component_id": "Position",
          "data": { "x": 10.0, "y": 20.0 }
        }
      }
    ]
  }
}
```

### Programmatic Access

You can also enable debug mode programmatically:

```rust
use tx2_link::init_debug_mode;

fn main() {
    // Reads TX2_DEBUG and TX2_TRACE environment variables
    init_debug_mode();

    // Your code here - debug logging happens automatically
}
```

### Why Debug Mode?

Binary protocols like MessagePack and Bincode are efficient but opaque. Without debug mode, inspecting what's being sent over the wire requires:
- Packet capture tools
- Manual deserialization
- Custom logging code

With debug mode, you get instant visibility into:
- What data is changing between snapshots
- How effective delta compression is
- Serialization format efficiency
- Rate limiting behavior
- Network traffic patterns

This makes tx2-link extremely "vibe coding friendly" - you can see exactly what's happening without writing debugging code.

## Development Status

- [x] Core protocol messages
- [x] Delta compression with field-level diffing
- [x] Multi-format serialization (MessagePack, Bincode, JSON)
- [x] Transport abstractions (WebSocket, IPC, stdio, memory)
- [x] Rate limiting (token bucket, sliding window)
- [x] Schema versioning and validation
- [x] Streaming serializer/deserializer
- [x] Comprehensive benchmarks
- [x] Debug mode with JSON logging and traces
- [ ] Compression (zstd/lz4) for large snapshots
- [ ] Encryption support
- [ ] Reconnection handling with state recovery
- [ ] Binary diff algorithms for large binary components

## Dependencies

- `serde` - Serialization framework
- `rmp-serde` - MessagePack format
- `bincode` - Bincode format
- `serde_json` - JSON format
- `tokio` - Async runtime
- `bytes` - Efficient byte buffers
- `ahash` - Fast hashing
- `thiserror` - Error handling

## License

MIT

## Contributing

Contributions are welcome! This is part of the broader TX-2 project for building isomorphic applications with a unified world model.

## Learn More

- [TX-2 Framework Outline](../frameworkoutline.md)
- [tx2-core Native Engine](../tx2-core)
- [tx2-pack Snapshot Format](../tx2-pack)
- [tx2-ecs TypeScript Runtime](https://github.com/IreGaddr/tx2-ecs)