Crate binserde

binserde

A crate similar to serde, but specialized for serializing into a compact binary format, including features like string deduplication.

This crate is very WIP. Features currently not implemented but planned include incremental versioning support so that old formats can still be loaded when the data format changes, deduplication of arbitrary data structures, and explicit tagging (writing a struct or enum as a set of key/value pairs instead of serializing the items in order of declaration, for higher resistance to format changes at the expense of output size)

Usage

use std::fs::File;
use std::io::BufReader;

#[derive(BinSerialize, BinDeserialize, Eq, PartialEq)]
struct MyData {
    v1: String,
    v2: Option<usize>,
}

let my_data = MyData {
    v1: "Some Text".to_string(),
    v2: Some(12415165),
};

let vec = binserde::serialize(&my_data).unwrap();

let copy_of_my_data: MyData = binserde::deserialize(&vec).unwrap();

assert_eq!(my_data, copy_of_my_data);

Macro Attributes

#[derive(BinSerialize)] and #[derive(BinDeserialize)] allows using attributes on the type itself and its fields to control (de)serialization.

#[binserde(skip)]

Valid for: fields

Skips the field when serializing. When deserializing, uses Default::default() instead of reading from the stream to fill the field.

#[binserde(no_dedup)]

Valid for: fields

Turns off deduplication for this field. See [Deduplication] for more information about how it works.

#[binserde(index = n)]

Valid for: fields

not implemented

Moves the field and all following fields to the specified position n when serializing, shifting everything originally after that position to the right.

Example:

#[derive(BinSerialize)]
struct S {
    w: u8,
    x: u8,
    #[binserde(index = 0)]
    y: u8,
    z: u8,
}

let vec = binserde::serialize(&S { w: 0, x: 1, y: 2, z: 3 });

assert_eq!(&[2, 3, 0, 1], &vec);

The attribute moved y and z into position 0, pushing w and x back to positions 2 and 3 respectively.

The attribute can be applied on more than one field, in which case moving operations will be evaluated from top to bottom. That means, the following struct serializes in the order z, x, y, w and not x, y, z, w or any other order:

#[derive(BinSerialize)]
struct S {
    w: u8,
    #[binserde(index = 0)]
    x: u8,
    y: u8,
    #[binserde(index = 0)]
    z: u8,
}

Deduplication

Deduplication is currently only implemented for strings. It works by taking any String or str that is serialized using its BinSerializer implementation and adds it to a seperate list which is written to the beginning of the buffer given to serialize (or an equivalent function, after which the actual data follows. In that data, the string is replaced by a usize pointing to the index in the string list. Effectively, a deduplicated data structure gets transformed from this:

struct S {
    s1: String,
    s2: String,
    strs: Vec<String>,
    something_else: u32,
}

to this:

struct S1 {
    strings: Vec<String>,
    s1: usize,
    s2: usize,
    strs: Vec<usize>,
    something_else: u32,
}

when serializing. This can have a major impact on the resulting size of the serialized data structure when multiple occurrences of the same string appear.

Re-exports

pub use de::BinDeserialize;

pub use de::BinDeserializer;

pub use ser::BinSerialize;

pub use ser::BinSerializer;

pub use serde::Mode;

Modules

de
dedup
ser
serde
try_iter
util

Enums

Error

Functions

deserialize
deserialize_from
deserialize_in_place
deserialize_with
deserialize_with_from
serialize
serialize_into
serialize_with
serialize_with_into

Type Definitions

Result

Derive Macros

BinDeserialize
BinSerialize