Crate binn_ir[−][src]

An implementation of Binn - https://github.com/liteserver/binn

Project

Features

  • All official types are supported.
  • User defined types are not yet supported.

Notes

  • Unless absolutely necessary, the project will not use any dependency.
  • #![no_std] might be supported when alloc crate is stabilized.
  • IR stands for implementation in Rust.
  • There will be no plan to support secure encoder/decoder via cryptography. The author considers that another field for experts. However, simple API for safe encoder/decoder will be supported. For example: option to limit container size to be decoded...

Quick examples

Most functionalities are provided via 2 traits: Encoder, Decoder.

This example demonstrates a simple file container:

extern crate binn_ir;

use binn_ir::value::{Value, Encoder, Decoder};

const MAGIC_NUMBER: u64 = 0xABCD;

// Buffer
let mut buf: Vec<u8> = vec![];

// Magic number
buf.encode_u64(MAGIC_NUMBER).unwrap();

// A single file header contains: name and hash
let file_header = {
    let mut map = std::collections::BTreeMap::new();
    map.insert(0_i32, Value::from("the-sun"));  // name
    map.insert(1_i32, Value::U64(0));           // hash
    map
};
let file_content = "is hot";

// Encode data (using ::clone() to use the variable later in assertions)
buf.encode_map(file_header.clone()).unwrap();
buf.encode_blob(file_content.as_bytes()).unwrap();

// Now decode data back
let mut cursor = std::io::Cursor::new(&buf);
assert_eq!(cursor.decode_u64().unwrap(), Some(MAGIC_NUMBER));
assert_eq!(cursor.decode_map().unwrap(), Some(file_header));
assert_eq!(cursor.decode_blob().unwrap().unwrap(), file_content.as_bytes());
assert_eq!(cursor.decode().unwrap(), None);
assert_eq!(cursor.position(), buf.len() as u64);

Thanks

Special thanks to Binn's authors for their hard work.

Copy of specification

Below is a copy of Binn's specification, with some minor changes of formatting.

Binn Specification

Format

Each value is stored with 4 possible parameters:

[type][size][count][data]

But most are optional. Only the type parameter is used in all of them. Here is a list of used parameters for basic data types:

boolean, null:
[type]

int, float (storage: byte, word, dword or qword):
[type][data]

string, blob:
[type][size][data]

list, object, map:
[type][size][count][data]

Each parameter can be stored with polymorphic size:

Parameter Size
[type] 1 or 2 bytes
[size] 1 or 4 bytes
[count] 1 or 4 bytes
[data] n bytes

[Type]

Each value is stored starting with the data type. It can use 1 or 2 bytes. The first byte is divided as follows:

 +-------- Storage type
 |  +----- Sub-type size
 |  |  +-- Sub-type
000 0 0000

Storage

The 3 most significant bits are used for the storage type. It has information about how many bytes the data will use. The storage type can be any of:

  • No additional bytes
  • 1 Byte
  • Word (2 bytes, big endian)
  • Dword (4 bytes, big endian)
  • Qword (8 bytes, big endian)
  • String (UTF-8, null terminated)
  • Blob
  • Container

And the constants are:

Storage Bits Hex Dec
NOBYTES 000 0 0000 0x00 0
BYTE 001 0 0000 0x20 32
WORD 010 0 0000 0x40 64
DWORD 011 0 0000 0x60 96
QWORD 100 0 0000 0x80 128
STRING 101 0 0000 0xA0 160
BLOB 110 0 0000 0xC0 192
CONTAINER 111 0 0000 0xE0 224

Sub-type size

The next bit informs if the type uses 1 or 2 bytes.

If the bit is 0, the type uses only 1 byte, and the sub-type has 4 bits (0 to 15)

 +-------- Storage type
 |  +----- Sub-type size
 |  |  +-- Sub-type
000 0 0000

When the bit is 1, another byte is used for the type and the sub-type has 12 bits (up to 4096)

 +-------- Storage type
 |  +----- Sub-type size
 |  |
000 1 0000  0000 0000
      |  Sub-type   |
      +-------------+

Sub-type

Each storage can have up to 4096 sub-types. They hold what kind of value is stored in that storage space.

Example: a DWORD can contain a signed integer, an unsigned integer, a single precision floating point number, and many more... even user defined types

Here are the values for basic data types, with the sub-type highlighted:

Type Storage Bits Hex Dec
Null NOBYTES 0000 0000 0x00 0
True NOBYTES 0000 0001 0x01 1
False NOBYTES 0000 0010 0x02 2
UInt8 BYTE 0010 0000 0x20 32
Int8 BYTE 0010 0001 0x21 33
UInt16 WORD 0100 0000 0x40 64
Int16 WORD 0100 0001 0x41 65
UInt32 DWORD 0110 0000 0x60 96
Int32 DWORD 0110 0001 0x61 97
Float DWORD 0110 0010 0x62 98
UInt64 QWORD 1000 0000 0x80 128
Int64 QWORD 1000 0001 0x81 129
Double QWORD 1000 0010 0x82 130
Text STRING 1010 0000 0xA0 160
DateTime STRING 1010 0001 0xA1 161
Date STRING 1010 0010 0xA2 162
Time STRING 1010 0011 0xA3 163
DecimalStr STRING 1010 0100 0xA4 164
Blob BLOB 1100 0000 0xC0 192
List CONTAINER 1110 0000 0xE0 224
Map CONTAINER 1110 0001 0xE1 225
Object CONTAINER 1110 0010 0xE2226

User Defined Types

An application can use a different DateTime type and store the value in a DWORD or QWORD.

Storage = QWORD (0x80)
Sub-type = 5 (0x05) [choose any unused]

Type DateTime = (0x80 | 0x05 => 0x85)

An application can send HTML inside a Binn structure and can define a type to differ from plain text.

Storage = STRING (0xA0)
Sub-type = 9 (0x09) [choose any unused]

Type HTML = (0xA0 | 0x09 => 0xA9)

If the sub-type is greater than 15, a new byte must be used, and the sub-type size bit must be set:

Storage = STRING (0xA000)
Sub-type size = (0x0100)
Sub-type = 21 (0x0015)

Type HTML = (0xA000 | 0x1000 | 0x0015 => 0xB015)

The created type parameter must be stored as big-endian.

[Size]

This parameter is used in strings, blobs and containters. It can have 1 or 4 bytes.

If the first bit of size is 0, it uses only 1 byte. So when the data size is up to 127 (0x7F) bytes the size parameter will use only 1 byte.

Otherwise a 4 byte size parameter is used, with the msb 1. Leaving us with a high limit of 2 GigaBytes (0x7FFFFFFF).

Data size Size Parameter Uses
<= 127 bytes 1 byte
> 127 bytes 4 bytes

There is no problem if a small size is stored using 4 bytes. The reader must accept both.

For strings, the size parameter does not include the null terminator.

For containers, the size parameter includes the type parameter. It stores the size of the whole structure.

[Count]

This parameter is used only in containers to inform the number of items inside them. It can have 1 or 4 bytes, formatted exactly as the size parameter.

Count Count Parameter Uses
<= 127 items 1 byte
> 127 items 4 bytes

Containers

List

Lists are containers that store values one after another.

The count parameter informs the number of values inside the container.

[123, "test", 2.5, true]

Map

Maps are associative arrays using integer numbers for the keys.

The keys are stored using a big-endian DWORD (4 bytes) that are read as a signed integer.

So the current limits are from INT32_MIN to INT32_MAX. But there is room for increase if needed.

The count parameter informs the number of key/value pairs inside the container.

{1: 10, 5: "the value", 7: true}

Object

Objects are associative arrays using text for the keys.

The keys are not null terminated and the limit is 255 bytes long.

The keys are stored preceded by the key length using a single byte for it.

The count parameter informs the number of key/value pairs inside the container.

{"id": 1, "name": "John", "points": 30.5, "active": true}

Limits

Type Min Max
Integers INT64_MIN UINT64_MAX
Floating point numbers IEEE 754
Strings 0 2 GB
Blobs 0 2 GB
Containers 4 2 GB

Associative Arrays

Key type Min Max
Number INT32_MIN INT32_MAX
Text 0 255 bytes

Sub-types: up to 4096 for each storage type

Example Structures

A json data such as {"hello":"world"} is serialized as:

Binn: (17 bytes)

  \xE2           // [type] object (container)
  \x11           // [size] container total size
  \x01           // [count] key/value pairs
  \x05hello      // key
  \xA0           // [type] = string
  \x05           // [size]
  world\x00      // [data] (null terminated)

A list of 3 integers:

Json: (14 bytes)

[123, -456, 789]

Binn: (11 bytes)

  \xE0           // [type] list (container)
  \x0B           // [size] container total size
  \x03           // [count] items
  \x20           // [type] = uint8
  \x7B           // [data] (123)
  \x41           // [type] = int16
  \xFE\x38       // [data] (-456)
  \x40           // [type] = uint16
  \x03\x15       // [data] (789)

A list inside a map:

Json: (25 bytes)

{1: "add", 2: [-12345, 6789]}

Binn: (26 bytes)

 \xE1             // [type] map (container)
 \x1A             // [size] container total size
 \x02             // [count] key/value pairs
 \x00\x00\x00\x01 // key
 \xA0             // [type] = string
 \x03             // [size]
 add\x00          // [data] (null terminated)
 \x00\x00\x00\x02 // key
 \xE0             // [type] list (container)
 \x09             // [size] container total size
 \x02             // [count] items
 \x41             // [type] = int16
 \xCF\xC7         // [data] (-12345)
 \x40             // [type] = uint16
 \x1A\x85         // [data] (6789)

A list of objects:

Json: (47 bytes)

[ {"id": 1, "name": "John"}, {"id": 2, "name": "Eric"} ]

Binn: (43 bytes)

 \xE0           // [type] list (container)
 \x2B           // [size] container total size
 \x02           // [count] items

 \xE2           // [type] object (container)
 \x14           // [size] container total size
 \x02           // [count] key/value pairs

 \x02id         // key
 \x20           // [type] = uint8
 \x01           // [data] (1)

 \x04name       // key
 \xA0           // [type] = string
 \x04           // [size]
 John\x00       // [data] (null terminated)

 \xE2           // [type] object (container)
 \x14           // [size] container total size
 \x02           // [count] key/value pairs

 \x02id         // key
 \x20           // [type] = uint8
 \x02           // [data] (2)

 \x04name       // key
 \xA0           // [type] = string
 \x04           // [size]
 Eric\x00       // [data] (null terminated)

Modules

storage

Storages
value

Values

Constants

CRATE_CODE_NAME

Crate code name
CRATE_NAME

Crate name
CRATE_RELEASE_DATE

Crate release date (year/month/day)
CRATE_VERSION

Crate version
TAG

Tag, which can be used for logging...
UUID

Unique universally identifier of this crate. Its CRC-32 is 149dc8a5.