Crate compact_encoding

Series of compact encoding schemes for building small and fast parsers and serializers

Binary compatible with the original JavaScript compact-encoding library.

Usage

The simplest way to encoded and decode a some data is using the to_encoded_bytes and map_decode macros:

use compact_encoding::{map_decode, to_encoded_bytes};

let number = 41_u32;
let word = "hi";

let encoded_buffer = to_encoded_bytes!(number, word);
let ((number_dec, word_dec), remaining_buffer) = map_decode!(&encoded_buffer, [u32, String]);

assert!(remaining_buffer.is_empty());
assert_eq!(number_dec, number);
assert_eq!(word_dec, word);

Manual encoding and decoding

When more fine-grained control of encoding and decoding is needed you manually do each step of encoding and decoding like in the following example, where we want to use a fixed width encoding for number (instead of the default variable width encoding). It shows how to manually calculate the needed buffer size, create the buffer, encode data, and decode it.

use compact_encoding::{CompactEncoding, FixedWidthEncoding, FixedWidthU32};

let number = 41_u32;
let word = "hi";

// Use `encoded_size` to figure out how big a buffer should be
let size = number.as_fixed_width().encoded_size()? + word.encoded_size()?;

// Create a buffer with the calculated size
let mut buffer = vec![0; size];
assert_eq!(buffer.len(), 4 + 1 + 2);

// Then actually encode the values
let mut remaining_buffer = number.as_fixed_width().encode(&mut buffer)?;
remaining_buffer = word.encode(remaining_buffer)?;
assert!(remaining_buffer.is_empty());
assert_eq!(buffer.to_vec(), vec![41_u8, 0, 0, 0, 2_u8, b'h', b'i']);

// `buffer` now contains all the encoded data, and we can decode from it
let (number_dec, remaining_buffer) = FixedWidthU32::decode(&buffer)?;
let (word_dec, remaining_buffer) = String::decode(remaining_buffer)?;
assert!(remaining_buffer.is_empty());
assert_eq!(number_dec, number);
assert_eq!(word_dec, word);

Implementing CompactEncoding for custom types

Here we demonstrate how to implement CompactEncoding for a custom struct.

use compact_encoding::{
    map_decode, map_encode, sum_encoded_size, to_encoded_bytes, CompactEncoding, EncodingError,
};

#[derive(Debug, PartialEq)]
struct MyStruct {
    some_flag: bool,
    values: Option<Vec<[u8; 32]>>,
    other: String,
    stuff: u64,
}

impl CompactEncoding for MyStruct {
    fn encoded_size(&self) -> Result<usize, EncodingError> {
        Ok(1 /* flags */ + {
             /* handle option values */
            if let Some(values) = &self.values  {
                values.encoded_size()?
            } else {
                0
            }
        } + sum_encoded_size!(&self.other, &self.stuff))
    }

    fn encode<'a>(&self, buffer: &'a mut [u8]) -> Result<&'a mut [u8], EncodingError> {
        let mut flags: u8 = 0;
        if self.some_flag {
            flags |= 1;
        }
        if self.values.is_some() {
            flags |= 2;
        }
        let mut rest = flags.encode(buffer)?;
        if let Some(values) = &self.values {
            rest = values.encode(rest)?;
        }
        Ok(map_encode!(rest, self.other, self.stuff))
    }

    fn decode(buffer: &[u8]) -> Result<(Self, &[u8]), EncodingError> {
        let (flags, rest) = u8::decode(buffer)?;
        let some_flag: bool = flags & 1 != 0;
        let (values, rest) = if flags & 2 != 0 {
            let (vec, rest) = <Vec<[u8; 32]>>::decode(rest)?;
            (Some(vec), rest)
        } else {
            (None, rest)
        };
        let ((other, stuff), rest) = map_decode!(rest, [String, u64]);
        Ok((
            Self {
                some_flag,
                values,
                other,
                stuff,
            },
            rest,
        ))
    }
}

// Test values
let foo = MyStruct {
    some_flag: false,
    values: None,
    other: "hi".to_string(),
    stuff: 42,
};

let bar = MyStruct {
    some_flag: true,
    values: Some(vec![[1; 32], [2; 32]]),
    other: "yo".to_string(),
    stuff: 0,
};

// Encode `foo` and `bar` to a buffer
let buffer = to_encoded_bytes!(&foo, &bar);

// With the above use of a flags byte, the empty value encodes to only one byte
assert_eq!(
    buffer.len(),
    // flags + string + u64
    (1 + 3 + 1) +
    // "" + (values.len().encoded_size() + (values.len() * <[u8;32]>::encoded_size()) + ""
    (1 + (1 + (2 * 32)) + 3 + 1)
);

// And decode directly to your own struct
let (foo_dec, rest) = MyStruct::decode(&buffer)?;
let (bar_dec, rest) = MyStruct::decode(&rest)?;
// Ensure all bytes were used
assert!(rest.is_empty());
assert_eq!(foo_dec, foo);
assert_eq!(bar_dec, bar);

Macros

create_buffer

Given a list of CompactEncoding things, create a zeroed buffer of the correct size for encoding. Note this is macro is useful when your arguments have differing types.

map_decode

Decode a buffer to the list of types provided, returning the remaining buffer. It takes as arguments: (&buffer, [type1, type2, type3, ...]) And returns: ((decoded_type1, decoded_type2, ...), remaining_buffer)

map_encode

Given a buffer and a list of CompactEncoding things, encode the arguments to the buffer. Note this is macro is useful when your arguments have differing types.

sum_encoded_size

Given a list of CompactEncoding things, sum the result of calling CompactEncoding::encoded_size on all of them. Note this is macro is useful when your arguments have differing types.

to_encoded_bytes

Given a list of CompactEncoding things, encode the arguments to the buffer. Note this is macro is useful when your arguments have differing types.

Structs

EncodingError

Encoding/decoding error.

FixedWidthUint

A fixed width encodable unsigned integer

Enums

EncodingErrorKind

Specific type EncodingError

Constants

U16_SIGNIFIER

indicates a variable width unsigned integer fits in u16

U32_SIGNIFIER

indicates a variable width unsigned integer fits in u32

U64_SIGNIFIER

indicates a variable width unsigned integer fits in u64

Traits

BoxedSliceEncodable

Define this trait for T to get impl Box<[T]> for CompactEncoding

CompactEncoding

A trait for building small and fast parsers and serializers.

FixedWidthEncoding

Implents functionality needed to encode unisegned integrer in a fixed width way with CompactEncoding

VecEncodable

Implement this for type T to have CompactEncoding implemented for Vec<T>

Functions

as_array

Get a slice as an array of size N. Errors when slice.len() != N.

as_array_mut

Get a slice as a mutable array of size N. Errors when slice.len() != N.

bytes_fixed_from_vec

Helper to convert a vec to an array, and fail with an encoding error when needed

decode_bytes_fixed

Decode a fixed sized array from a buffer. Return the array and the remainder of the buffer. Errors when buffer.len() < N;

decode_usize

Decode a usize from buffer and return the remaining bytes. This will fail, when we are decoding a usize on a usize = u32 machine for data that was originally encoded on a usize = u64 machine whenever the value is over u32::MAX.

encode_bytes_fixed

Encoded a fixed sized array to a buffer, return the remainder of the buffer. Errors when value.len() > buffer.len();

encode_var_u64

Write uint to the start of buffer and return the remaining part of buffer.

encoded_size_usize

The number of bytes required to encode this number. Note this is always variable width.

encoded_size_var_u64

The number of bytes required to encode this number. We only need this for u64 because all other uints can be converted to usize reliably.

get_slices_checked

Split a slice in two at mid. Returns encoding error when mid is out of bounds.

get_slices_mut_checked

Split a mutable slice into two mutable slices around mid. Returns encoding error when mid is out of bounds.

take_array

split the first N bytes of buffer off and return them

take_array_mut

split the first N bytes of buffer off and return them

write_array

Write source to buffer and return the remainder of buffer. Errors when N < buffer.len()

write_slice

write source to buffer and return remaining buffer

Type Aliases

FixedWidthU32

A wrapper around u32 to let us encoded/decode to/from a fixed width

FixedWidthU64

A wrapper around u64 to let us encoded/decode to/from a fixed width