Crate msgpacker 

MessagePacker - a no-std msgpack implementation

The protocol specification can be found here.

This crate targets simplicity and performance. No dependencies are used, just the standard Rust library.

It will implement Packable and Unpackable for Rust atomic types. The traits can also be implemented manually.

Features

• alloc: Implements the functionality for Vec, String, and unlocks custom extensions.
• derive: Enables MsgPacker derive convenience macro.
• strict: Will panic if there is a protocol violation of the size of a buffer; the maximum allowed size is u32::MAX.
• std: Will implement the Packable and Unpackable for std collections.
• serde: Adds support for serde

Non-uniform collections

MessagePack is a language-agnostic format. Dynamically typed languages like Python, JavaScript, and Ruby naturally allow mixed-type collections — for instance, a Python list [0, 1694166331209.0] containing both an integer and a float is perfectly valid. When these values are serialized into MessagePack, the resulting byte stream encodes each element with its own type tag (u64, f64, etc.), producing an array whose elements have heterogeneous types.

Rust’s type system does not directly support such collections: a Vec<T> requires a single concrete T. As noted in #18, the native Packable/Unpackable traits cannot deserialize these non-uniform arrays because they rely on a statically known element type at compile time.

The serde feature provides a workaround: deserialize the MessagePack bytes into serde_json::Value, which is a dynamically typed enum that can represent any JSON-compatible value. This will incur performance overhead compared to the native traits, since serde uses a visitor pattern that involves runtime type dispatch and heap allocations for every element.

use msgpacker::serde;
use serde_json::Value;

// MessagePack bytes encoding a 2-element array: [0_u64, 1694166331209.0_f64]
// This kind of payload is common when receiving data from Python, JS, or other
// dynamically typed languages that don't distinguish collection element types.
let bytes: &[u8] = &[146, 0, 203, 66, 120, 167, 66, 234, 244, 144, 0];

// Deserialize into a dynamic Value — works for any valid MessagePack payload
let value: Value = serde::from_slice(bytes).unwrap();
let items = value.as_array().unwrap();

// Each element retains its original type
assert!(items[0].is_u64());
assert!(items[1].is_f64());

If your use case involves only uniform collections (e.g. Vec<u64>), prefer the native Packable/Unpackable traits for zero-overhead deserialization.

Example

use msgpacker::prelude::*;
use std::collections::HashMap;

// boilerplate derives - those aren't required
#[derive(Debug, PartialEq, Eq)]
// this convenience derive macro will implement `Packable` and `Unpackable`
#[derive(MsgPacker)]
pub struct City {
    name: String,

    // The traits are implemented for stdlib collections. If you have a custom map, you can use the
    // directive `#[msgpacker(map)]` so the traits will be automatically implemented through the
    // iterators of the map.
    inhabitants_per_street: HashMap<String, u64>,

    // This is also automatically implemented. The manual implementation is via `#[msgpacker(array)]`.
    zones: Vec<String>,
}

// create an instance of a city.
let city = City {
    name: "Kuala Lumpur".to_string(),
    inhabitants_per_street: HashMap::from([
        ("Street 1".to_string(), 10),
        ("Street 2".to_string(), 20),
    ]),
    zones: vec!["Zone 1".to_string(), "Zone 2".to_string()],
};

// serialize the city into bytes
let mut buf = Vec::new();
let n = city.pack(&mut buf);
println!("serialized {} bytes", n);

// deserialize the city and assert correctness
let (n, deserialized) = City::unpack(&buf).unwrap();
println!("deserialized {} bytes", n);
assert_eq!(city, deserialized);

Serde

Version 0.5.0 introduces serde support.

use msgpacker::serde;
use serde_json::{json, Value};

let val = serde_json::json!({"foo": "bar"});
let ser = serde::to_vec(&val);
let des: Value = serde::from_slice(&ser).unwrap();

assert_eq!(val, des);

While it’s important to recognize that serde’s performance can be notably slower, this is primarily due to its implementation of a visitor pattern for type serialization, rather than solely relying on the static structure of declarations. However, serde is broadly used and having its support is helpful since a plethora of other libraries will be automatically supported just by having this feature enabled.

For more information, refer to Benchmarks.

Benchmarks

Results obtained with AMD EPYC 7402P 24-Core Processor.

	msgpacker	msgpacker serde	rmps
pack 1	814.26 ns	146.61 µs	191.35 µs
pack 10	7.5772 µs	905.06 µs	1.2513 ms
pack 1000	4.8276 ms	88.910 ms	116.45 ms
unpack 1	1.0011 µs	164.48 µs	270.51 µs
unpack 10	11.017 µs	1.1243 ms	1.6636 ms
unpack 1000	14.546 ms	112.81 ms	164.89 ms

To run the benchmarks:

cd msgpacker-bench && cargo bench

Note on Github

Although GitHub offers exceptional CI and hosting services virtually for free, its questionable approach towards user sovereignty and privacy is notable. Consequently, I chose to disengage from their infrastructure.

For more information, check Give Up GitHub!

Modules

prelude

Required types for the library.

serde

serde implementations.

Enums

Error

Deserialization errors for the protocol implementation.

Extension

Custom extension definition as reference to a bytes source.

Traits

Packable

A packable type.

Unpackable

An unpackable type.

Functions

pack_array

Packs an array into the extendable buffer, returning the amount of written bytes.

pack_map

Packs a map into the extendable buffer, returning the amount of written bytes.

pack_to_vec

Packs the provided packable value into a vector.

unpack_array

Unpacks an array from the buffer, returning a collectable type and the amount of read bytes.

unpack_array_iter

Unpacks an array from the iterator, returning a collectable type and the amount of read bytes.

unpack_map

Unpacks a map from the buffer, returning a collectable type and the amount of read bytes.

unpack_map_iter

Unpacks a map from the iterator, returning a collectable type and the amount of read bytes.

Derive Macros

MsgPacker