Crate serser 

Serser – the Rust serialization engine

serser is a serialization/deserialization framework for Rust. It uses a stream of tokens (e.g. U32, Seq, and EndSeq) as the intermediate representation.

Examples

Rust To JSON

use serser::derive::*;                // The derive macros.
use serser::json::json_from_tokens;
use serser::prelude::*;               // Includes the {From,Into]Tokens traits.

#[derive(FromTokens, IntoTokens)]
struct A {
  b: u32
}

let original = A { b: 42 };
let mut json = String::new();
json_from_tokens(&mut json, original)?;

assert_eq!(r#"{"b":42}"#, json);

JSON To Rust

use serser::derive::*;                // The derive macros.
use serser::json::json_into;
use serser::prelude::*;               // Includes the {From,Into]Tokens traits.

#[derive(Debug, Eq, PartialEq)]       // For the assert_eq.
#[derive(FromTokens, IntoTokens)]
struct A {
  b: u32
}

let got = json_into::<A>(r#"{ "b": 42 }"#)?;

assert_eq!(got, A { b: 42 });

Rust To Rust

This is emulating Clone:

use serser::derive::*;                // The derive macros.
use serser::prelude::*;               // Includes the {From,Into]Tokens traits.
use serser::TokenVec;

#[derive(Debug, Eq, PartialEq)]       // For the assert_eq.
#[derive(FromTokens, IntoTokens)]
struct A {
  b: u32
}

let mut tokens = TokenVec::new();     // A buffer, useful for testing.

let original = A { b: 42 };
original.into_tokens(&mut tokens)?;   // Writes to `tokens`.
let clone = A::from_tokens(tokens)?;  // Reads from `tokens`.

assert_eq!(clone, original);

Compared To serde

The serde crate is the defacto standard for serialization, both for JSON and as used in wasm-bindgen for communicating with the host from WebAssembly.

This crate aims to improve on the serde design in these ways:

• A smaller API, making it easier to write custom (de)serializers. Only one function needs to be implemented for a sink: yield_token. Another function can be overridden to provide hints for the next expected token.
• A bufferable intermediate representation that can be shared between threads, or stored for later use. Primarily lightweight, reference-based, Token objects are used. An OwningToken is provided for easy storage and IPC.
• A push-based pipeline, inverting the Iterator pattern to allow correct lifetimes to be expressed more easily. Tokens only live during the yield_token invocation; if the callee needs to store it, it has to make a copy. This is similar to serde, but avoids the ping-pong between Deserializer, *Access and Visitor.
• No reliance on 'static for metadata like struct field names.

The aim is for the implementation to be on a par with serde in terms of performance.

The name serser comes from the idea that serialization and deserialization are fundamentally the same thing, just switching perspectives of the source and sink. It is a conversion from one representation to another, through a common data model.

Processing Model

This shows a flow where a recursive data structure like [42] is being parsed. The FromTokens trait is a convenience wrapper around FromTokenSink. It has a blanket implementation for all FromTokenSink.

The FromTokenSink acts as a per-type TokenSink factory. It is implemented for most Rust types.

The IntoTokens trait drives the whole chain. It calls TokenSink::yield_token for every token, and may use TokenSink::expect_tokens to negotiate data types with the sink. The diagram below only shows one lane per trait, but the calls may go to different implementations. It may recursively call other IntoTokens.

The TokenSink is responsible for handling the sinks. This may be writing to a std::fmt::Write, or writing into a Rust data structure buffer. The FromTokenSink::from_sink function knows how to extract the buffered value from a particular TokenSink implementation. A sink may use FromTokenSink to off-load handling of constituent pieces of data to another sink. It owns the sinks it creates, forwards tokens to them, meaning that IntoTokens is oblivious to any subsinks. The TokenSink::expect_tokens function can use FromTokenSink::expect_initial to figure out what a nested data type requires, without creating a sink.

   Trait               Trait              Trait             Trait
FromTokens         FromTokenSink       IntoTokens         TokenSink
-----------------------------------------------------------------------
from_tokens
             --> new_sink
            <..
             ------------------------> into_tokens
                                                   ---> yield_token
                                                   ---> (expect_tokens)
                 (expect_initial) <-------------------
                                   ...................>
                 new_sink         <-------------------
                                   ...................>
                 from_sink        <-------------------
                                   ...................>
                                                   ---> ...
                                           |
                                           V
                                       into_tokens
                                                   ---> yield_token
                                                   ---> (expect_tokens)
                 (expect_initial) <-------------------
                                                   ---> ...
                                           :
                                           V
            <........................
             --> from_sink
            <..

Modules

derive

The derive macro that automatically implements the IntoTokens for your structs and enums.

json

Parses and serializes JSON.

prelude

A module that puts the useful traits into scope.

test

Utilities for testing and debugging.

token

Definitions of the tokens and metadata.

Structs

TokenVec

A vector of tokens, which can be used as a source or sink.

Enums

TokenError

A simple Error implementation for when nothing special is needed.

Traits

Error

Errors returned from TokenSink must implement this trait.

FromTokenSink

Provides a way to associate appropriate sinks with Rust data types.

FromTokens

Constructs a Rust value from a token source.

IntoTokens

Provides a way to associate appropriate sources with Rust data types.

TokenSink

A receiver of tokens. This forms the basis for a push-parser. The source side calls Self::yield_token for each simple token. Nested structures are handled by the sink, i.e. any subsink required to parse a data structure is owned by the parent sink, which forwards tokens.

Functions

iter_into_tokens

Yields the contents of an iterator as a Token::Seq.

port

Uses FromTokens and IntoTokens to port from source to sink, using a stream of Token as intermediary representation.