musli

Müsli

Müsli is a flexible and generic binary serialization framework.

Müsli currently depends on GATs and is nightly-only

We make the following assumptions:

Anything being deserialized must be fully held in memory and able to hand out contiguous slices of it. This allows users of musli to perform zero-copy deserialization for bytes-oriented types.
Decoding is biased to assume strings are encoded verbatim in the format used so that references to strings can always be used. That means strings have to be UTF-8. A format that deviates from this will have to rely on runtime errors.

I've chosen to internally use the term "encoding", "encode", and "decode" because it's common terminology when talking about binary formats. It's also distinct from serde's use of "serialization" allowing for the ease of using both libraries side by side if desired.

Design

Müsli is designed with similar principles as serde. Relying on Rust's powerful trait system to generate code which can largely be optimized away. The end result should be very similar to a handwritten encoder / decoder.

The central components of the framework are the Encode and Decode derives. They are thoroughly documented in the derives module.

Usage

Add the following to your Cargo.toml:

musli = "0.0.8"
musli-wire = "0.0.8"

Formats

Formats are currently distinguished by supporting various degrees of upgrade stability. A fully upgrade stable encoding format must tolerate that one model can add fields that an older version of the model should be capable of ignoring.

Partial upgrade stability can still be useful as is the case of the musli-storage format below, because reading from storage only requires decoding to be upgrade stable. So if correctly managed with #[musli(default)] this will never result in any readers seeing unknown fields.

The available formats and their capabilities are:

	reorder?	missing?	unknown?
musli-storage `#[musli(packed)]`	✗	✗	✗
musli-storage	✔	✔	✗
musli-wire	✔	✔	✔

recorder? determines whether fields must occur in exactly the order in which they are specified. So reordering fields in such a struct would cause an error. This is only suitable for byte-oriented IPC where data models are strictly synchronized.

missing? determines if the reader can handle missing fields, as exemplified above. This is suitable for on-disk storage.

unknown? determines if the format can skip over unknown fields. This is suitable for network communication.

For every feature you drop, the format becomes more compact and efficient. musli-storage #[musli(packed)] for example is roughly as compact and efficient as bincode while musli-wire is comparable to something like protobuf*.

Examples

The following is an example of full upgrade stability using musli-wire:

use musli::{Encode, Decode};

#[derive(Debug, PartialEq, Encode, Decode)]
struct Version1 {
    name: String,
}

#[derive(Debug, PartialEq, Encode, Decode)]
struct Version2 {
    name: String,
    #[musli(default)]
    age: Option<u32>,
}

let version2 = musli_wire::to_vec(&Version2 {
    name: String::from("Aristotle"),
    age: Some(62),
})?;

let version1: Version1 = musli_wire::decode(&version2[..])?;

assert_eq!(version1, Version1 {
    name: String::from("Aristotle"),
});

The following is an example of partial upgrade stability using musli-storage:

use musli::{Encode, Decode};

let version2 = musli_storage::to_vec(&Version2 {
    name: String::from("Aristotle"),
    age: Some(62),
})?;

assert!(musli_storage::decode::<_, Version1>(&version2[..]).is_err());

let version1 = musli_storage::to_vec(&Version1 {
    name: String::from("Aristotle"),
})?;

let version2: Version2 = musli_storage::decode(&version1[..])?;

assert_eq!(version2, Version2 {
    name: String::from("Aristotle"),
    age: None,
});