Alkahest - Fantastic serialization library.

Alkahest is blazing-fast, zero-deps, zero-overhead, zero-unsafe, schema-based serialization library. It is suitable for broad range of use-cases, but tailored for custom high-performance network protocols.

Benchmarks

This benchmark that mimics some game networking protocol.

See also benchmark results from https://github.com/djkoloski/rust_serialization_benchmark (in draft until 0.2 release).

Features

• Schema-based serialization. Alkahest uses data schemas called Formulas to serialize and deserialize data. Thus controlling data layout independently from data types that are serialized or deserialized.

• Support wide variety of formulas. Integers, floats, booleans, tuples, arrays, slices, strings and user-defined formulas with custom data layout using derive macro that works for structs and enums of any complexity and supports generics.

• Zero-overhead serialization of sequences. Alkahest support serializing iterators directly into slice formulas. No more allocation of a Vec to serialize and drop immediately.

• Lazy deserialization. Alkahest provides Lazy<F> type to deserialize any formula F lazily. Lazy can be used later to perform actual deserialization.
  Lazy<[F]> can also produce iterator that deserializes elements on demand.
  Laziness is controlled on type level and can be applied to any element of a larger formula.

• Infallible serialization. Given large enough or growing buffer any value that implements Serialize can be serialized without error. No more unnecessary unwraps or puzzles "what to do if serialization fails?". The only error condition for serialization is "data doesn't fit".

Planned features

• Serializable formula descriptors
• Compatibility rules
• External tool for code-generation for formula descriptors for C and Rust.

How it works. In more details

Alkahest separates data schema definition (aka Formula) from serialization and deserialization code. Doing so, this library provides better guarantees for cases when serializable data type and deserializable data type are different. It also supports serializing from iterators instead of collections and deserialization into lazy wrappers that defers costly process and may omit it entirely if value is never accessed. User controls laziness on type level by choosing appropriate Deserialize impls. For instance deserializing into Vec<T> is eager because Vec<T> is constructed with all T instances and memory allocated for them. While DeIter obtained from Lazy<[F]> implements Iterator and deserializes elements in Iterator::next and other methods. And provides constant-time random access to any element for sized formulas.

Flexibility comes at cost of using only byte slices for serialization and deserialization. And larger footprint of serialized data than some other binary formats.

Question about support of dense data packing is open. It may be desireable to control on type level.

Errors and panics

The API is designed with following principles: Any value can be serialized successfully given large enough buffer. Data can't cause panic, incorrect implementation of a trait can.

There is zero unsafe code in the library on any code it generates. No UB is possible given that std is not unsound.

Forward and backward compatibility

No data schemas stays the same. New fields and variants are added, others are deprecated and removed.

There's set of rules that ensures forward compatibility between formulas. And another set or rules for backward compatibility.

Verification of compatibility is not implemented yet.

Forward compatibility

Forward compatibility is an ability to deserialize data that was serialized into newer formulas.

TODO: List all rules

Backward compatibility

Backward compatibility is an ability to deserialize data that was serialized into older formulas.

TODO: List all rules

Formula, Serialize and Deserialize traits.

The crate works using three fundamental traits. Formula, Serialize and Deserialize. There's also supporting trait - BareFormula.

Alkahest provides derive macros for Formula, Serialize and Deserialize.

Formula

Formula trait is used to allow types to serve as data schemas. Any value serialized into given formula should be deserializable from the same formula. Sharing only Formula type allows modules and crates easily communicate. Formula dictates binary data layout and it must be platform-independent.

Potentially Formula types can be generated from separate files, opening possibility for cross-language communication.

Formula is implemented for a number of types out-of-the-box. Primitive types like [bool], integers and floating point types all implement Formula. This excludes isize and usize. In their place there's [FixedUsize] and [FixedIsize] types provided, whose size is controlled by a feature-flag. !Caveat!: Sizes and addresses are serialized as [FixedUsize]. Truncating usize value if it was too large. This may result in broken data generated and panic in debug. Increase size of the [FixedUsize] if you encounter this. It is also implemented for tuples, array and slice, Option and Vec (the later requires "alloc" feature).

The easiest way to define a new formula is to derive Formula trait for a struct or an enum. Generics are supported, but may require complex bounds specified in attributes for Serialize and Deserialize derive macros. The only constrain is that all fields must implement Formula.

Serialize

Serialize<F> trait is used to implement serialization according to a specific formula F. Serialization writes to mutable bytes slice and should not perform any dynamic allocations. Binary result of any type serialized into a formula must follow it. At the end, if a stream of primitives serialized is the same, binary result should be the same. Types may be serializable into different formulas producing different binary result.

Serialize is implemented for many types. Most notably there's implementation T: Serialize<T> and &T: Serialize<T> for all primitives T (except usize and isize). Another important implementation is Serialize<F> for I where I: IntoIterator, I::Item: Serialize<F>, allowing serializing into slice directly from both iterators and collections. Serialization with formula Ref<F> uses serialization into formula F and then stores relative address and size. No dynamic allocations is required.

Deriving Serialize for a type will generate Serialize implementation, formula is specified in attribute #[alkahest(FormulaRef)] or #[alkahest(serialize(FormulaRef))]. FormulaRef is typically a type. When generics are used it also contains generic parameters and bounds. If formula is not specified - Self is assumed. Formula should be derived for the type as well. It is in-advised to derive Serialize for formulas with manual Formula implementation, Serialize derive macro generates code that uses non-public items generated by Formula derive macro. So either both should have manual implementation or both derived.

For structures Serialize derive macro requires that all fields are present on both Serialize and Formula structure and has the same order (trivially if this is the same structure).

For enums Serialize derive macro checks that for each variant there exists variant on Formula enum. Variants content is compared similar to structs. Serialization inserts variant ID and serializes variant as struct. The size of variants may vary. Padding is inserted by outer value serialization if necessary.

Serialize can be derived for structure where Formula is an enum. In this case variant should be specified using #[alkahest(@variant_ident)] or #[alkahest(serialize(@variant_ident))] and then Serialize derive macro will produce serialization code that works as if this variant was a struct Formula, except that variant's ID will be serialized before fields.

Serialize can be derived for enum only if Formula is enum as well. Serializable enum may omit some (or all) variants from Formula. It may not have variants missing in Formula. Each variant then follows rules for structures.

For convenience Infallible implements Serialize for enum formulas.

Deserialize

Deserialize<'de, F> trait is used to implement deserialization according to a specific formula. Deserialization reads from bytes slice constructs deserialized value. Deserialization should not perform dynamic allocations except those that required to construct and initialize deserialized value. E.g. it is allowed to allocate when [Vec<T>] is produced if non-zero number of T values are deserialized. It should not over-allocate.

Similar to Serialize alkahest provides a number of out-of-the-box implementations of Deserialize trait. From<T> types can be deserialized from primitive formula T.

Values that can be deserialized from formula F can also deserialize from Ref<F>, it reads address and length and proceeds with formula F.

Vec<T> may deserialize from slice formula. Deserialize<'de, [F]> is implemented for Lazy<'de, [F]> type that may produce DeIter<'de, F, T> - an Iterator that lazily deserialize elements of type T: Deserialize<'de, F>. DeIter is cloneable, can be iterated from both ends and skips elements in constant time for sized formulas.

Deriving Deserialize for a type will generate Deserialize implementation, formula is specified in attribute #[alkahest(FormulaRef)] or #[alkahest(deserialize(<formula-ref>))]. <formula-ref> is typically a type. When generics are used it also contains generic parameters and bounds. If formula is not specified - Self is assumed. Formula should be derived for the type as well. It is in-advised to derive Deserialize for formulas with manual Formula implementation, Deserialize derive macro generates code that uses non-public items generated by Formula derive macro. So either both should have manual implementation or both derived.

Interoperability with serde

Alkahest is cool but serde is almost universally used, and for good reasons. While designing a Formula it may be desireable to include existing type that supports serialization serde, especially if it comes from another crate. This crate provides Bincode and Bincoded<T> formulas to cover this. Anything with serde::Serialize implementation can be serialized into Bincode formula, naturally it will be serialized using bincode crate. Bincoded<T> is a restricted version of Bincode that works only for T.

Usage example

// This requires two default features - "alloc" and "derive".
#[cfg(all(feature = "derive", feature = "alloc"))]
fn main() {
  use alkahest::{Formula, Serialize, Deserialize, serialize_to_vec, deserialize};

  // Define simple formula. Make it self-serializable.
  #[derive(Clone, Debug, PartialEq, Eq, Formula, Serialize, Deserialize)]
  struct MyDataType {
    a: u32,
    b: Vec<u8>,
  }

  // Prepare data to serialize.
  let value = MyDataType {
    a: 1,
    b: vec![2, 3],
  };

  // Use infallible serialization to `Vec`.
  let mut data = Vec::new();

  // Note that this value can be serialized by reference.
  // This is default behavior for `Serialized` derive macro.
  // Some types required ownership transfer for serialization.
  // Notable example is iterators.
  let size = serialize_to_vec::<MyDataType, _>(&value, &mut data);

  let (de, de_size) = deserialize::<MyDataType, MyDataType>(&data).unwrap();
  assert_eq!(de_size, size);
  assert_eq!(de, value);
}

#[cfg(not(all(feature = "derive", feature = "alloc")))]
fn main() {}

License

Licensed under either of

• Apache License, Version 2.0, (license/APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (license/MIT or http://opensource.org/licenses/MIT)

at your option.

Contributions

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.