Crate shmp

Expand description

§shmp: A flexible and efficient MessagePack implementation

shmp offers a robust and easy-to-use toolkit for serializing and deserializing data with the MessagePack format in Rust. It is designed with serde as a first-class citizen, providing maximum performance and type safety.

§Two Ways to Work with MessagePack

There are two primary approaches to using shmp:

Direct Serialization with Serde (Recommended): Serialize your Rust structs directly to and from MessagePack bytes. This is the most efficient and idiomatic method.
Dynamic Pack Enum: Use the shmp::Pack enum, which is similar to serde_json::Value, to represent and manipulate MessagePack data when the structure is not known at compile time.

§Quick Start: Using Serde

The most common use case is to derive Serialize and Deserialize on your types and use the top-level to_vec and from_slice functions.

use serde::{Serialize, Deserialize};
use shmp::{to_vec, from_slice};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct User {
    id: u32,
    username: String,
    is_active: bool,
}

fn main() -> Result<(), shmp::Error> {
    let user = User {
        id: 1024,
        username: "shmp_user".to_string(),
        is_active: true,
    };

    // Serialize the struct into MessagePack bytes.
    let encoded: Vec<u8> = to_vec(&user)?;

    // Deserialize the bytes back into a struct.
    let decoded: User = from_slice(&encoded)?;

    assert_eq!(user, decoded);
    Ok(())
}

§Comprehensive Examples

This document provides a wide range of examples demonstrating how to use shmp for various serialization and deserialization tasks.

§1. Direct Serialization with `serde`

This is the most efficient and common way to use shmp.

§Basic Struct

use serde::{Serialize, Deserialize};
use shmp::{to_vec, from_slice, to_writer, from_reader};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct User {
    id: u32,
    name: String,
}

let user = User { id: 1, name: "shmp".to_string() };

// Serialize to a byte vector and back.
let encoded = to_vec(&user).unwrap();
let decoded: User = from_slice(&encoded).unwrap();
assert_eq!(user, decoded);

// You can also serialize to any `io::Write` stream.
let mut buffer = Vec::new();
to_writer(&mut buffer, &user).unwrap();

// And deserialize from any `io::Read` stream.
let mut cursor = std::io::Cursor::new(&buffer);
let decoded_from_reader: User = from_reader(&mut cursor).unwrap();
assert_eq!(user, decoded_from_reader);

§Enum Variants

shmp correctly serializes all styles of enum variants according to serde’s data model.

use serde::{Serialize, Deserialize};
use shmp::{to_vec, from_slice};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
enum Message {
    Quit, // Unit variant -> "Quit"
    Write(String), // Newtype variant -> {"Write": "Hello"}
    Move(i32, i32), // Tuple variant -> {"Move": [10, 20]}
    Update { id: u64, value: String }, // Struct variant -> {"Update": {"id": 1, "value": "..."}}
}

let quit = Message::Quit;
let write = Message::Write("Hello".to_string());
let mv = Message::Move(10, 20);
let update = Message::Update { id: 1, value: "new".to_string() };

let encoded = to_vec(&quit).unwrap();
let decoded: Message = from_slice(&encoded).unwrap();
assert_eq!(decoded, quit);

let encoded = to_vec(&write).unwrap();
let decoded: Message = from_slice(&encoded).unwrap();
assert_eq!(decoded, write);

let encoded = to_vec(&mv).unwrap();
let decoded: Message = from_slice(&encoded).unwrap();
assert_eq!(decoded, mv);

let encoded = to_vec(&update).unwrap();
let decoded: Message = from_slice(&encoded).unwrap();
assert_eq!(decoded, update);

§Zero-Copy Deserialization

For maximum efficiency, you can deserialize string slices (&str) and byte slices (&[u8]) without allocating new memory. This requires using the #[serde(borrow)] attribute.

Note that zero-copy deserialization is only available when using from_slice, as it requires the entire byte slice to be available in memory to borrow from.

use serde::{Serialize, Deserialize};
use shmp::{to_vec, from_slice};

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct Packet<'a> {
    #[serde(borrow)]
    command: &'a str,
    #[serde(borrow, with = "serde_bytes")] // This is essential for &[u8]
    payload: &'a [u8],
}

let original = Packet {
    command: "SET",
    payload: &[0xDE, 0xAD, 0xBE, 0xEF],
};

let encoded = to_vec(&original).unwrap();

// The `decoded` struct borrows its data directly from the `encoded` slice.
let decoded: Packet = from_slice(&encoded).unwrap();

assert_eq!(original, decoded);

§Custom Extension (`Ext`) Types

You can serialize your custom types into the MessagePack Ext format by implementing EncodeExt/DecodeExt and using #[serde(with = "shmp::ext_format")].

#[derive(Debug, PartialEq)]
struct MyCustomType {
    id: u32,
    value: String,
}

impl EncodeExt for MyCustomType {
    const EXT_TAG: i8 = 42;
    fn to_ext_data(&self) -> Result<Vec<u8>, Error> {
        let mut bytes = self.id.to_be_bytes().to_vec();
        bytes.extend_from_slice(self.value.as_bytes());
        Ok(bytes)
    }
}

impl DecodeExt for MyCustomType {
    const EXT_TAG: i8 = 42;
    fn from_ext_data(data: &[u8]) -> Result<Self, Error> {
        // Note: In production code, you should implement robust error handling
        //       to prevent panics from slicing or `unwrap()`, such as checking `data` length.
        //       This example is simplified for clarity.
        let id = u32::from_be_bytes(data[0..4].try_into().unwrap());
        let value = String::from_utf8(data[4..].to_vec())?;
        Ok(MyCustomType { id, value})
    }
}

#[derive(Serialize, Deserialize, Debug, PartialEq)]
struct DataPacket {
    sequence: u64,
    #[serde(with = "shmp::ext_format")]
    payload: MyCustomType,
}

let packet = DataPacket {
    sequence: 1,
    payload: MyCustomType { id: 101, value: "hello".to_string() },
};

let encoded = to_vec(&packet).unwrap();
let decoded: DataPacket = from_slice(&encoded).unwrap();
assert_eq!(packet, decoded);

§Using Ext Directly (Low-level)

In addition to the trait-based approach using #[serde(with = "...")], you can use the Ext struct to manipulate extension types directly at a low level. This is useful when you want to send tagged binary data without needing to define a full custom type.


// Manually create an Ext value with a custom tag (e.g., 77) and payload.
let original_ext = Ext(77, vec![10, 20, 30, 40]); 

// It serializes like any other Serde-compatible type, but is specifically
// encoded into the MessagePack `ext` format, not a `map` like a typical struct.
let encoded = to_vec(&original_ext).unwrap(); 
assert_eq!(encoded[0], 0xd6_u8); // Verify it's a fixext 4 type
let decoded_ext: Ext = from_slice(&encoded).unwrap();
assert_eq!(original_ext, decoded_ext);

// You can then access the tag and data directly.
assert_eq!(decoded_ext.0, 77);
assert_eq!(decoded_ext.1, vec![10, 20, 30, 40]);

§2. Dynamic `Pack` Enum

The Pack enum is useful when the data structure is not known at compile time. The pack! macro provides a convenient JSON-like syntax.

§Creating `Pack` values with `pack!`

use shmp::{pack, Pack};

// simple examples
assert_eq!(pack!(null), Pack::Nil);
assert_eq!(pack!(true), Pack::Boolean(true));
assert_eq!(pack!(1004), Pack::Int(1004));
assert_eq!(pack!("Hello"), Pack::String("Hello".to_string()));
assert_eq!(pack!(1000 + 4), Pack::Int(1004));
assert_eq!(pack!(if true { 1000 } else { 4 }), Pack::Int(1000));
assert_eq!(pack!(({ let x = 1004; x })), Pack::Int(1004));
assert_eq!(pack!(block { let x = 1004; x }), Pack::Int(1004));

// compound type example
let data = pack!({
    "user_id": 1024,
    "username": "shmp_user",
    "tags": ["rust", "msgpack", "serde"],
    "config": {
        "retries": 3,
        "timeout": null
    }
});

assert_eq!(data["username"], pack!("shmp_user"));
assert_eq!(data["tags"][0], pack!("rust"));
assert_eq!(data["config"]["timeout"], pack!(null));

§Interchangeability: Struct <-> Pack <-> bytes

shmp supports seamless conversion between strongly-typed structs, the dynamic Pack enum, and raw bytes. No matter which path you choose (e.g., Struct -> bytes or Struct -> Pack -> bytes), the resulting MessagePack output is identical. This flexibility allows you to select the best approach for your needs. Use direct struct conversion for performance and type-safety when the schema is known, or leverage the Pack enum as an intermediate representation for dynamic data manipulation.

#[derive(Serialize, Deserialize, Debug, PartialEq, Clone)]
struct Config {
    theme: String,
    features: Vec<String>,
}

let config = Config {
    theme: "dark".to_string(),
    features: vec!["A".to_string(), "B".to_string()],
};

// --- Serialization Path ---
// You can serialize a struct to bytes directly, or via an intermediate `Pack` value.
// The resulting bytes will be identical.

// 1. Struct -> bytes
let bytes_direct = to_vec(&config).unwrap();

// 2. Struct -> Pack -> bytes
let pack_value = to_pack(&config).unwrap();
let bytes_via_pack = to_vec(&pack_value).unwrap();

assert_eq!(bytes_direct, bytes_via_pack);

// --- Deserialization Path ---
// Similarly, you can deserialize bytes directly to a struct, or via `Pack`.

// 1. bytes -> Struct
let decoded_direct: Config = from_slice(&bytes_direct).unwrap();

// 2. bytes -> Pack -> Struct
let pack_from_bytes: Pack = from_slice(&bytes_direct).unwrap();
let decoded_via_pack: Config = from_pack(pack_from_bytes).unwrap();

assert_eq!(config, decoded_direct);
assert_eq!(decoded_direct, decoded_via_pack);

Modules§

ext_format: Provides a generic module for use with #[serde(with = "...")] to serialize and deserialize custom types as the MessagePack Ext format.

Macros§

pack: Creates a shmp::Pack value using a JSON-like syntax.

Structs§

Ext: A low-level struct for directly handling MessagePack’s ext format, representing a (type, data) pair.
MpSerializer: A Serializer for the MessagePack format.
PackDeserializer: A serde::Deserializer that deserializes a Pack enum value into a Rust value.
PackSerializer: A serde::Serializer that serializes Rust values into the Pack enum.
SliceDeserializer: A Deserializer for the MessagePack format from a byte slice.
StreamDeserializer: A Deserializer for the MessagePack format.

Enums§

Error: The error type for shmp operations, covering I/O, serialization, and data conversion issues.
Pack: A self-describing, dynamic MessagePack value.

Traits§

DecodeExt: A trait for types that can be decoded from the data part of a MessagePack Ext format.
EncodeExt: A trait for types that can be encoded into the data part of a MessagePack Ext format.

Functions§

from_pack: Converts a Pack enum value into an instance of type T.
from_reader: A convenience function to deserialize a Rust value from a generic io::Read.
from_slice: A convenience function to deserialize a Rust value from a &[u8] slice.
read_pack: Reads a single MessagePack value from a reader and decodes it into a Pack enum.
to_pack: Converts a value that implements serde::Serialize into a Pack enum value.
to_vec: A convenience function to serialize a Rust value into a Vec<u8>.
to_writer: A convenience function to serialize a Rust value to a generic io::Write.
write_pack: Encodes a Pack enum into its MessagePack binary representation and writes it to a writer.

Crate shmp

Crate shmp Copy item path

§shmp: A flexible and efficient MessagePack implementation

§Two Ways to Work with MessagePack

§Quick Start: Using Serde

§Comprehensive Examples

§1. Direct Serialization with serde

§Basic Struct

§Enum Variants

§Zero-Copy Deserialization

§Custom Extension (Ext) Types

§Using Ext Directly (Low-level)

§2. Dynamic Pack Enum

§Creating Pack values with pack!

§Interchangeability: Struct <-> Pack <-> bytes

Modules§

Macros§

Structs§

Enums§

Traits§

Functions§

Crate shmp

§1. Direct Serialization with `serde`

§Custom Extension (`Ext`) Types

§2. Dynamic `Pack` Enum

§Creating `Pack` values with `pack!`