async-io-typed

Documentation

Combines bincode and futures to adapt any AsyncRead or AsyncWrite type into a channel for transmission of serde compatible Rust types.

Who needs this?

If you have an endpoint you need to communicate with and

You can establish some kind of I/O connection to it (i.e. TCP, named pipes, or a unix socket)
You need clear message boundaries
You're not trying to conform to an existing wire format such as HTTP or protobufs. This crate uses a custom format.
The data you wish to send can be easily represented in a Rust type, and that type implements serde's Deserialize and Serialize traits.

Then this crate might be useful to you!

Who doesn't need this?

If the endpoint is in the same process then you should not use this crate. You're better served by existing async mpsc channels. Many crates provide async mpsc channels, including futures and tokio. Pick your favorite implementation. Additionally, if you're trying to interface with a process that doesn't have Rust code, and can't adopt a Rust portion, this crate will hurt much more than it will help. Consider using protobufs or JSON if Rust adoption is a blocker.

Binary format

Introduction

The main offering of this crate is a consistent and known representation of Rust types. As such, the format is considered to be part of our stable API, and changing the format requires a major version number bump. To aid you in debugging, that format is documented here.

High-level overview

The byte stream is split up into messages. Every message begins with a length value. After length bytes have been read, a new message can begin immediately afterward. This length value is the entirety of the header of a message. Messages have no footer. The bytes read out of a message are then deserialized into a Rust type via bincode, using the following configuration

bincode::DefaultOptions::new()
    .with_limit(size_limit)
    .with_little_endian()
    .with_varint_encoding()
    .reject_trailing_bytes()

Length encoding

The length is encoded using a variably sized integer encoding scheme. To understand this scheme, first we need a few constant values.

u16_marker; decimal: 252, hex: FC
u32_marker; decimal: 253, hex: FD
u64_marker; decimal: 254, hex: FE
zst_marker; decimal: 255, hex: FF
stream_end; decimal: 0,   hex: 00

Any length less than u16_marker and greater than 0 is encoded as a single byte whose value is the length. A length of zero is encoded with the zst_marker. The stream is ended with the stream_end value. When this is read the peer is expected to close the connection.

async-io-typed always uses little-endian. The user data being sent may contain values that are not little-endian, but async-io-typed itself always uses little-endian.

If the first byte is u16_marker, then the length is 16 bits wide, and encoded in the following 2 bytes. Once those 2 bytes are read, the message begins. u32_marker and u64_marker are used in a similar way, each of those being 4 bytes, and 8 bytes respectively.

Examples

Length 12

0C

Length 0

FF

Length 252 (First byte is u16_marker)

FC, FC, 00

Length 253 (First byte is u16_marker)

FC, FD, 00

Length 65,536 (aka 2^16) (First byte is u32_marker)

FD, 00, 00, 01, 00

Length 4,294,967,296 (aka 2^32) (First byte is u64_marker)

FE, 00, 00, 00, 00, 01, 00, 00, 00

async-io-typed 1.0.4