MessagePacker - a no-std msgpack implementation

The protocol specification can be found here.

This crate targets simplicity and performance. It extends bytes BufMut, for increased interoperability.

The library is designed with production in mind, and will return auxiliary, useful runtime data such as buffer consumption bytes count.

Example

use msgpacker::{MsgPacker, Packable as _, Unpackable as _};
use std::collections::HashMap;

// boilerplate derives - those aren't required
#[derive(Debug, PartialEq, Eq)]
// this convenience derive macro will implement `Packable` and `Unpackable`
#[derive(MsgPacker)]
pub struct City {
    name: String,

    // The traits are implemented for stdlib collections. If you have a custom map, you can use the
    // directive `#[msgpacker(map)]` so the traits will be automatically implemented through the
    // iterators of the map.
    inhabitants_per_street: HashMap<String, u64>,

    // This is also automatically implemented. The manual implementation is via `#[msgpacker(array)]`.
    zones: Vec<String>,

    // Binary serialization is specialized in MessagePack. We can use the directive here.
    #[msgpacker(binary)]
    metadata: Vec<u8>,
}

// create an instance of a city.
let city = City {
    name: "Kuala Lumpur".to_string(),
    inhabitants_per_street: HashMap::from([
        ("Street 1".to_string(), 10),
        ("Street 2".to_string(), 20),
    ]),
    zones: vec!["Zone 1".to_string(), "Zone 2".to_string()],
    metadata: b"foo".to_vec(),
};

// serialize the city into bytes
let buf = city.pack_to_vec();

// deserialize the city and assert correctness
let deserialized = City::unpack(&buf).unwrap();
assert_eq!(city, deserialized);

We also support borrowed deserialization:

use msgpacker::{MsgPackerBorrowed, Packable as _, UnpackableBorrowed as _};

#[derive(Debug, MsgPackerBorrowed, PartialEq, Eq)]
pub struct City<'a> {
    name: &'a str,
}

// create an instance of a city.
let city = City {
    name: "Kuala Lumpur"
};

let buf = city.pack_to_vec();

let deserialized = City::unpack(&buf).unwrap();
assert_eq!(city, deserialized);

Features

• derive: Enables MsgPacker derive convenience macro.
• strict: Will panic if there is a protocol violation of the size of a buffer; the maximum allowed size is u32::MAX.
• assertions: Will panic on runtime if an invalid buffer is provided for prealloc serialization.
• std: Will implement the Packable and Unpackable for std collections.
• serde: Adds support for serde

Serde

Version 0.5.0 introduces serde support.

use msgpacker::serde;
use serde_json::{json, Value};

let val = serde_json::json!({"foo": "bar"});
let ser = serde::to_vec(&val);
let des: Value = serde::from_slice(&ser).unwrap();

assert_eq!(val, des);

Serde performance is excelling for deserialization, but is slower for serialization.

For more information, refer to Benchmarks.

Benchmarks

Results obtained with AMD EPYC 7402P 24-Core Processor.

The results have their correctness asserted for each benchmark item.

• pack alloc: creates a Vec for each serialization. Useful if the user doesn't know the size of his serialized instances. It's the most common case.
• pack prealloc: preallocates a Vec with the known serialization length, and executes the packing with a mutable, no-alloc slice.
• unpack owned: unpacks the structure to an owned type. Useful for complex collections such as HashMap.
• unpack borrowed: unpacks the structure as a borrowed type from the original bytes. Useful for very simple structures with mostly atomic types suck as Vec<_> and String (not supported by rmp-serde).

To run the benchmarks:

RUSTFLAGS="-C target-cpu=native" cargo bench --profile bench

Non-uniform collections

MessagePack is a language-agnostic format. Dynamically typed languages like Python, JavaScript, and Ruby naturally allow mixed-type collections — for instance, a Python list [0, 1694166331209.0] containing both an integer and a float is perfectly valid. When these values are serialized into MessagePack, the resulting byte stream encodes each element with its own type tag (u64, f64, etc.), producing an array whose elements have heterogeneous types.

Rust's type system does not directly support such collections: a Vec<T> requires a single concrete T. As noted in #18, the native Packable/Unpackable traits cannot deserialize these non-uniform arrays because they rely on a statically known element type at compile time.

The serde feature provides a workaround: deserialize the MessagePack bytes into serde_json::Value, which is a dynamically typed enum that can represent any JSON-compatible value. This will incur performance overhead compared to the native traits, since serde uses a visitor pattern that involves runtime type dispatch and heap allocations for every element.

use msgpacker::serde;
use serde_json::Value;

// MessagePack bytes encoding a 2-element array: [0_u64, 1694166331209.0_f64]
// This kind of payload is common when receiving data from Python, JS, or other
// dynamically typed languages that don't distinguish collection element types.
let bytes: &[u8] = &[146, 0, 203, 66, 120, 167, 66, 234, 244, 144, 0];

// Deserialize into a dynamic Value — works for any valid MessagePack payload
let value: Value = serde::from_slice(bytes).unwrap();
let items = value.as_array().unwrap();

// Each element retains its original type
assert!(items[0].is_u64());
assert!(items[1].is_f64());

If your use case involves only uniform collections (e.g. Vec<u64>), prefer the native Packable/Unpackable traits for zero-overhead deserialization.

Note on Github

Although GitHub offers exceptional CI and hosting services virtually for free, its questionable approach towards user sovereignty and privacy is notable. Consequently, I chose to disengage from their infrastructure.

For more information, check Give Up GitHub!