Crate msgpacker 

MessagePacker - a no-std msgpack implementation

The protocol specification can be found here.

This crate targets simplicity and performance. It extends bytes BufMut, for increased interoperability.

The library is designed with production in mind, and will return auxiliary, useful runtime data such as buffer consumption bytes count.

Example

use msgpacker::{MsgPacker, Packable as _, Unpackable as _};
use std::collections::HashMap;

// boilerplate derives - those aren't required
#[derive(Debug, PartialEq, Eq)]
// this convenience derive macro will implement `Packable` and `Unpackable`
#[derive(MsgPacker)]
pub struct City {
    name: String,

    // The traits are implemented for stdlib collections. If you have a custom map, you can use the
    // directive `#[msgpacker(map)]` so the traits will be automatically implemented through the
    // iterators of the map.
    inhabitants_per_street: HashMap<String, u64>,

    // This is also automatically implemented. The manual implementation is via `#[msgpacker(array)]`.
    zones: Vec<String>,

    // Binary serialization is specialized in MessagePack. We can use the directive here.
    #[msgpacker(binary)]
    metadata: Vec<u8>,
}

// create an instance of a city.
let city = City {
    name: "Kuala Lumpur".to_string(),
    inhabitants_per_street: HashMap::from([
        ("Street 1".to_string(), 10),
        ("Street 2".to_string(), 20),
    ]),
    zones: vec!["Zone 1".to_string(), "Zone 2".to_string()],
    metadata: b"foo".to_vec(),
};

// serialize the city into bytes
let buf = city.pack_to_vec();

// deserialize the city and assert correctness
let deserialized = City::unpack(&buf).unwrap();
assert_eq!(city, deserialized);

We also support borrowed deserialization:

use msgpacker::{MsgPackerBorrowed, Packable as _, UnpackableBorrowed as _};

#[derive(Debug, MsgPackerBorrowed, PartialEq, Eq)]
pub struct City<'a> {
    name: &'a str,
}

// create an instance of a city.
let city = City {
    name: "Kuala Lumpur"
};

let buf = city.pack_to_vec();

let deserialized = City::unpack(&buf).unwrap();
assert_eq!(city, deserialized);

Features

• derive: Enables MsgPacker derive convenience macro.
• strict: Will panic if there is a protocol violation of the size of a buffer; the maximum allowed size is u32::MAX.
• assertions: Will panic on runtime if an invalid buffer is provided for prealloc serialization.
• std: Will implement the Packable and Unpackable for std collections.
• serde: Adds support for serde

Serde

Version 0.5.0 introduces serde support.

use msgpacker::serde;
use serde_json::{json, Value};

let val = serde_json::json!({"foo": "bar"});
let ser = serde::to_vec(&val);
let des: Value = serde::from_slice(&ser).unwrap();

assert_eq!(val, des);

Serde performance is excelling for deserialization, but is slower for serialization.

For more information, refer to Benchmarks.

Benchmarks

Results obtained with AMD EPYC 7402P 24-Core Processor.

The results have their correctness asserted for each benchmark item.

• pack alloc: creates a Vec for each serialization. Useful if the user doesn’t know the size of his serialized instances. It’s the most common case.
• pack prealloc: preallocates a Vec with the known serialization length, and executes the packing with a mutable, no-alloc slice.
• unpack owned: unpacks the structure to an owned type. Useful for complex collections such as HashMap.
• unpack borrowed: unpacks the structure as a borrowed type from the original bytes. Useful for very simple structures with mostly atomic types suck as Vec<_> and String (not supported by rmp-serde).

instances count	msgpacker	msgpacker serde	rmps	zerompk
pack alloc 1	67.629 ns	251.75 ns	322.38 ns	277.77 ns
pack alloc 10	333.36 ns	764.16 ns	1.0981 µs	799.47 ns
pack alloc 100	3.6478 µs	4.2132 µs	6.9683 µs	4.4160 µs
pack alloc 1000	66.081 µs	67.055 µs	105.28 µs	66.911 µs
pack prealloc 1	49.170 ns	54.865 ns	94.714 ns	46.801 ns
pack prealloc 10	293.30 ns	367.28 ns	647.95 ns	264.44 ns
pack prealloc 100	2.9884 µs	3.5127 µs	6.2381 µs	2.8508 µs
pack prealloc 1000	56.887 µs	65.492 µs	102.09 µs	52.176 µs
unpack owned 1	137.10 ns	92.482 ns	251.77 ns	138.14 ns
unpack owned 10	1.2414 µs	1.0115 µs	2.2232 µs	992.32 ns
unpack owned 100	11.820 µs	9.2322 µs	22.659 µs	9.9914 µs
unpack owned 1000	231.64 µs	205.06 µs	333.20 µs	221.33 µs
unpack borrowed 1	96.822 ns	57.465 ns	†	67.606 ns
unpack borrowed 10	868.06 ns	662.06 ns	†	572.57 ns
unpack borrowed 100	7.9413 µs	5.8026 µs	†	5.4400 µs
unpack borrowed 1000	127.29 µs	101.97 µs	†	102.96 µs

To run the benchmarks:

RUSTFLAGS="-C target-cpu=native" cargo bench --profile bench

Non-uniform collections

MessagePack is a language-agnostic format. Dynamically typed languages like Python, JavaScript, and Ruby naturally allow mixed-type collections — for instance, a Python list [0, 1694166331209.0] containing both an integer and a float is perfectly valid. When these values are serialized into MessagePack, the resulting byte stream encodes each element with its own type tag (u64, f64, etc.), producing an array whose elements have heterogeneous types.

Rust’s type system does not directly support such collections: a Vec<T> requires a single concrete T. As noted in #18, the native Packable/Unpackable traits cannot deserialize these non-uniform arrays because they rely on a statically known element type at compile time.

The serde feature provides a workaround: deserialize the MessagePack bytes into serde_json::Value, which is a dynamically typed enum that can represent any JSON-compatible value. This will incur performance overhead compared to the native traits, since serde uses a visitor pattern that involves runtime type dispatch and heap allocations for every element.

use msgpacker::serde;
use serde_json::Value;

// MessagePack bytes encoding a 2-element array: [0_u64, 1694166331209.0_f64]
// This kind of payload is common when receiving data from Python, JS, or other
// dynamically typed languages that don't distinguish collection element types.
let bytes: &[u8] = &[146, 0, 203, 66, 120, 167, 66, 234, 244, 144, 0];

// Deserialize into a dynamic Value — works for any valid MessagePack payload
let value: Value = serde::from_slice(bytes).unwrap();
let items = value.as_array().unwrap();

// Each element retains its original type
assert!(items[0].is_u64());
assert!(items[1].is_f64());

If your use case involves only uniform collections (e.g. Vec<u64>), prefer the native Packable/Unpackable traits for zero-overhead deserialization.

Note on Github

Although GitHub offers exceptional CI and hosting services virtually for free, its questionable approach towards user sovereignty and privacy is notable. Consequently, I chose to disengage from their infrastructure.

For more information, check Give Up GitHub!

Modules

serde

serde implementations.

Structs

Encoder

An optimized encoder for dynamic allocation.

EncoderSlice

An optimized encoder for pre-allocated bytes.

Enums

Error

Deserialization errors for the protocol implementation.

Extension

Custom extension definition as reference to a bytes source.

Traits

BufMut

A trait for values that provide sequential write access to bytes.

Packable

A packable type.

Unpackable

An unpackable type.

UnpackableBorrowed

A borrowed unpackable type.

Functions

pack_array

Packs an array into the extendable buffer, returning the amount of written bytes.

pack_array_slice

Packs an array into the extendable buffer, returning the amount of written bytes.

pack_bytes

Packs a byte slice as msgpack binary, returning the amount of written bytes.

pack_bytes_option

Packs Option<&[u8]> as msgpack nil (None) or binary (Some), returning bytes written.

pack_map

Packs a map into the extendable buffer, returning the amount of written bytes.

pack_to_vec

Packs the provided packable value into a vector.

unpack_array

Unpacks an array from the buffer, returning a collectable type and the amount of read bytes.

unpack_array_borrowed

Unpacks an array from the buffer using the borrowed deserialization path.

unpack_array_iter

Unpacks an array from the iterator, returning a collectable type and the amount of read bytes.

unpack_bytes

Unpack bytes on binary format

unpack_bytes_iter

Unpack bytes on binary format using an iterator implementation.

unpack_bytes_option

Unpacks msgpack nil as None, or binary data as Some(Vec<u8>).

unpack_bytes_option_iter

Iterator-based version: unpacks nil as None, binary data as Some(Vec<u8>).

unpack_bytes_option_ref

Unpacks msgpack nil as None, or binary data as Some(&[u8]).

unpack_map

Unpacks a map from the buffer, returning a collectable type and the amount of read bytes.

unpack_map_iter

Unpacks a map from the iterator, returning a collectable type and the amount of read bytes.

Derive Macros

MsgPacker

MsgPackerBorrowed

MsgUnpackerBorrowed