pack-io 0.3.0

Why pack-io

Existing crates each cover a slice of the problem; none of them own all three properties together.

* rkyv requires alignment discipline at every use site; the cost is paid by the caller, not the codec.

The 1.0 contract is the same wire format on every supported platform, the same bytes for the same value every time, and no panic / no unbounded allocation on any input.

What it does

• Compact binary encoding of Rust values into a small, predictable wire format
• Schema-versioned messages - producer and consumer can be at different revisions and still interoperate
• Zero-copy deserialization for &[u8] / &str / length-prefixed slices when the input lives long enough
• Deterministic output - the same value always produces the same bytes (canonical encoding)
• Safe under untrusted input - bounded allocation, length-prefix validation, no panics on malformed bytes
• Runtime-agnostic - synchronous codec, usable from any context

Features

• Compact — small, fixed-overhead encoding; varint integers; length-prefixed byte slices
• Schema evolution — additive field changes, optional fields, version negotiation
• Zero-copy decode — view types that borrow from the input buffer where possible
• Deterministic — canonical encoding for hashing, signing, content-addressing
• Safe defaults — bounded allocation, validated lengths, no panics on bad input
• no_std-capable — embedded and constrained environments
• Derive macro — #[derive(Serialize, Deserialize)] for any struct or enum (feature derive)
• Optional serde interop — read and write serde types via a thin adapter (feature serde)

Roadmap snapshot

The roadmap is followed strictly; phases are not skipped. Per-phase exit criteria are tracked internally and surfaced in each release note.

Installation

[dependencies]
pack-io = "0.3"

# With derive macro (planned for 0.4+):
pack-io = { version = "0.3", features = ["derive"] }

# no_std build:
pack-io = { version = "0.3", default-features = false }

API surface (v0.3.0)

The full Tier-1 / Tier-2 / Tier-3 surface is live, with the wire format frozen for the 1.x line. See docs/API.md for the complete reference and docs/WIRE_FORMAT.md for the normative byte-level spec.

Tier 1 — the lazy path

use pack_io::{encode, decode};

let bytes: Vec<u8> = encode(&(7_u64, true, String::from("hello"))).unwrap();
let back: (u64, bool, String) = decode(&bytes).unwrap();
assert_eq!(back, (7, true, String::from("hello")));

Tier 2a — the in-memory Encoder / Decoder

Re-use a single Vec<u8> across many encodes; read several values from one buffer. Configuration (Config::max_alloc) is validated at construction time, not on every operation.

use pack_io::{Encoder, Decoder, Config};

let mut enc = Encoder::new();
enc.write(&7_u64).unwrap();
enc.write(&"hello").unwrap();
let bytes = enc.into_inner();

let cfg = Config::new().with_max_alloc(16 * 1024);
let mut dec = Decoder::with_config(&bytes, cfg).unwrap();
let n: u64 = dec.read().unwrap();
let s: String = dec.read().unwrap();
assert_eq!((n, s.as_str()), (7, "hello"));

Tier 2b — the streaming IoEncoder<W> / IoDecoder<R> (new in v0.3)

Write directly into any std::io::Write, read from any std::io::Read. Gated on the default std feature.

use pack_io::{IoEncoder, IoDecoder, encode_into, decode_from};
use std::io::Cursor;

// Single-shot helpers (encode_into / decode_from) over any Read / Write:
let mut sink: Vec<u8> = Vec::new();
encode_into(&(42_u64, "hello"), &mut sink).unwrap();
let back: (u64, String) = decode_from(&mut Cursor::new(sink)).unwrap();
assert_eq!(back, (42, "hello".to_string()));

// Or hold an encoder / decoder for multi-value streams:
let mut buf: Vec<u8> = Vec::new();
{
    let mut enc = IoEncoder::new(&mut buf);
    enc.write(&1_u64).unwrap();
    enc.write(&2_u64).unwrap();
}
let mut dec = IoDecoder::new(Cursor::new(buf));
let a: u64 = dec.read().unwrap();
let b: u64 = dec.read().unwrap();
assert_eq!((a, b), (1, 2));

Tier 3 — implement [Serialize] / [Deserialize] on your own types

Both traits are generic over the new Encode / Decode behaviour traits, so one impl works through every encoder flavour the crate ships (in-memory and streaming).

use pack_io::{Decode, Deserialize, Encode, Result, Serialize};

struct Point { x: i32, y: i32 }

impl Serialize for Point {
    fn serialize<E: Encode + ?Sized>(&self, enc: &mut E) -> Result<()> {
        self.x.serialize(enc)?;
        self.y.serialize(enc)
    }
}

impl Deserialize for Point {
    fn deserialize<D: Decode + ?Sized>(dec: &mut D) -> Result<Self> {
        Ok(Point {
            x: i32::deserialize(dec)?,
            y: i32::deserialize(dec)?,
        })
    }
}

The derive macro lands in 0.4.

Types supported in v0.3.0

Canonical map / set encoding (the determinism contract)

Hash-based collections (HashMap, HashSet) are encoded with entries sorted lexicographically by their encoded key bytes. A HashMap and a BTreeMap holding the same logical data therefore encode to identical bytes, regardless of insertion order or build-flag-dependent hash randomisation. This is the load-bearing property for hashing, signing, and content-addressing pack-io payloads. Full normative spec: docs/WIRE_FORMAT.md §4.

Invariants (held from v0.1.0)

• Round-trip integrity — decode(encode(v)) == v for every supported type, under any input.
• Determinism — the same value always produces the same bytes; no map-iteration-order leaks, no time-dependence, no platform-dependence.
• Safe decode — no panic, no unbounded allocation, no read past input, on any byte sequence.
• Wire-format stability — frozen at 1.0; any 1.x decoder reads any 1.x-or-earlier encoding.

These invariants hold for every release in the 0.x series. As of 0.3.0 they are enforced by 177 tests: round-trip + determinism property tests for every primitive and every collection (including the load-bearing "HashMap and BTreeMap encode identically" property), plus adversarial-decode harnesses that fuzz every public decode entry point with random bytes, truncations, and hostile length prefixes. The wire format itself is frozen for the 1.x line as of this release. A cargo-fuzz harness lands in 0.7.

Testing

# Stable + MSRV (1.85) on Linux / macOS / Windows, full feature matrix
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo test --no-default-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

# Supply chain
cargo audit
cargo deny check

# Concurrency model checking (decoders are stateless; loom coverage is light-touch)
RUSTFLAGS="--cfg loom" cargo test --test loom_codec

# Microbenchmarks
cargo bench --bench codec_bench

Examples

Each example is self-contained and runs against the published API of the version it was added in.

cargo run --example basic_roundtrip --release   # Tier-1 encode/decode of a tuple
cargo run --example primitive_tour --release    # one encoded value per primitive type
cargo run --example reuse_buffer --release      # Tier-2 Encoder + multi-value Decoder
cargo run --example collections_tour --release  # Vec / HashMap / BTreeMap / sets, canonical encoding
cargo run --example streaming_io --release      # IoEncoder / IoDecoder to a file

Cross-Platform Support

Tier 1 Support:

• ✅ Linux (x86_64, aarch64)
• ✅ macOS (x86_64, Apple Silicon)
• ✅ Windows (x86_64)

Encoding is byte-deterministic across all three; the CI matrix runs every target on stable and MSRV. Platform-specific behaviour is forbidden in the codec — there is no #[cfg(target_os = …)] branch on the encode or decode path.

Where It Fits

pack-io is the serialization substrate under network-protocol, wire-codec, and Hive DB. It is consumed by raft-io for log entries and by event-stream (when it lands) for message framing. It stays foreign-compatible: it works on its own without any other crate in the family.

Contributing

Before opening a PR, the full local checklist must pass:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo deny check
cargo audit

Any change touching the wire format requires a proptest round-trip and a determinism test in the same commit. Wire-format-breaking changes are not accepted after 0.3 without an accompanying migration note in CHANGELOG.md.