Crate compact_encoding

Expand description

§Series of compact encoding schemes for building small and fast parsers and serializers

Binary compatible with the original JavaScript compact-encoding library.

§Usage

The simplest way to encoded and decode a some data is using the to_encoded_bytes and map_decode macros:

use compact_encoding::{map_decode, to_encoded_bytes};

let number = 41_u32;
let word = "hi";

let encoded_buffer = to_encoded_bytes!(number, word);
let ((number_dec, word_dec), remaining_buffer) = map_decode!(&encoded_buffer, [u32, String]);

assert!(remaining_buffer.is_empty());
assert_eq!(number_dec, number);
assert_eq!(word_dec, word);

§Manual encoding and decoding

When more fine-grained control of encoding and decoding is needed you manually do each step of encoding and decoding like in the following example, where we want to use a fixed width encoding for number (instead of the default variable width encoding). It shows how to manually calculate the needed buffer size, create the buffer, encode data, and decode it.

use compact_encoding::{CompactEncoding, FixedWidthEncoding, FixedWidthU32};

let number = 41_u32;
let word = "hi";

// Use `encoded_size` to figure out how big a buffer should be
let size = number.as_fixed_width().encoded_size()? + word.encoded_size()?;

// Create a buffer with the calculated size
let mut buffer = vec![0; size];
assert_eq!(buffer.len(), 4 + 1 + 2);

// Then actually encode the values
let mut remaining_buffer = number.as_fixed_width().encode(&mut buffer)?;
remaining_buffer = word.encode(remaining_buffer)?;
assert!(remaining_buffer.is_empty());
assert_eq!(buffer.to_vec(), vec![41_u8, 0, 0, 0, 2_u8, b'h', b'i']);

// `buffer` now contains all the encoded data, and we can decode from it
let (number_dec, remaining_buffer) = FixedWidthU32::decode(&buffer)?;
let (word_dec, remaining_buffer) = String::decode(remaining_buffer)?;
assert!(remaining_buffer.is_empty());
assert_eq!(number_dec, number);
assert_eq!(word_dec, word);

§Implementing CompactEncoding for custom types

Here we demonstrate how to implement CompactEncoding for a custom struct.

use compact_encoding::{
    map_decode, map_encode, sum_encoded_size, to_encoded_bytes, CompactEncoding, EncodingError,
};

#[derive(Debug, PartialEq)]
struct MyStruct {
    some_flag: bool,
    values: Option<Vec<[u8; 32]>>,
    other: String,
    stuff: u64,
}

impl CompactEncoding for MyStruct {
    fn encoded_size(&self) -> Result<usize, EncodingError> {
        Ok(1 /* flags */ + {
             /* handle option values */
            if let Some(values) = &self.values  {
                values.encoded_size()?
            } else {
                0
            }
        } + sum_encoded_size!(&self.other, &self.stuff))
    }

    fn encode<'a>(&self, buffer: &'a mut [u8]) -> Result<&'a mut [u8], EncodingError> {
        let mut flags: u8 = 0;
        if self.some_flag {
            flags |= 1;
        }
        if self.values.is_some() {
            flags |= 2;
        }
        let mut rest = flags.encode(buffer)?;
        if let Some(values) = &self.values {
            rest = values.encode(rest)?;
        }
        Ok(map_encode!(rest, self.other, self.stuff))
    }

    fn decode(buffer: &[u8]) -> Result<(Self, &[u8]), EncodingError> {
        let (flags, rest) = u8::decode(buffer)?;
        let some_flag: bool = flags & 1 != 0;
        let (values, rest) = if flags & 2 != 0 {
            let (vec, rest) = <Vec<[u8; 32]>>::decode(rest)?;
            (Some(vec), rest)
        } else {
            (None, rest)
        };
        let ((other, stuff), rest) = map_decode!(rest, [String, u64]);
        Ok((
            Self {
                some_flag,
                values,
                other,
                stuff,
            },
            rest,
        ))
    }
}

// Test values
let foo = MyStruct {
    some_flag: false,
    values: None,
    other: "hi".to_string(),
    stuff: 42,
};

let bar = MyStruct {
    some_flag: true,
    values: Some(vec![[1; 32], [2; 32]]),
    other: "yo".to_string(),
    stuff: 0,
};

// Encode `foo` and `bar` to a buffer
let buffer = to_encoded_bytes!(&foo, &bar);

// With the above use of a flags byte, the empty value encodes to only one byte
assert_eq!(
    buffer.len(),
    // flags + string + u64
    (1 + 3 + 1) +
    // "" + (values.len().encoded_size() + (values.len() * <[u8;32]>::encoded_size()) + ""
    (1 + (1 + (2 * 32)) + 3 + 1)
);

// And decode directly to your own struct
let (foo_dec, rest) = MyStruct::decode(&buffer)?;
let (bar_dec, rest) = MyStruct::decode(&rest)?;
// Ensure all bytes were used
assert!(rest.is_empty());
assert_eq!(foo_dec, foo);
assert_eq!(bar_dec, bar);

Macros§

create_buffer: Given a list of CompactEncoding things, create a zeroed buffer of the correct size for encoding. Note this is macro is useful when your arguments have differing types.
map_decode: Decode a buffer to the list of types provided, returning the remaining buffer. It takes as arguments: (&buffer, [type1, type2, type3, ...]) And returns: ((decoded_type1, decoded_type2, ...), remaining_buffer)
map_encode: Given a buffer and a list of CompactEncoding things, encode the arguments to the buffer. Note this is macro is useful when your arguments have differing types.
sum_encoded_size: Given a list of CompactEncoding things, sum the result of calling CompactEncoding::encoded_size on all of them. Note this is macro is useful when your arguments have differing types.
to_encoded_bytes: Given a list of CompactEncoding things, encode the arguments to the buffer. Note this is macro is useful when your arguments have differing types.

Structs§

EncodingError: Encoding/decoding error.
FixedWidthUint: A fixed width encodable unsigned integer

Enums§

EncodingErrorKind: Specific type EncodingError

Constants§

U16_SIGNIFIER: indicates a variable width unsigned integer fits in u16
U32_SIGNIFIER: indicates a variable width unsigned integer fits in u32
U64_SIGNIFIER: indicates a variable width unsigned integer fits in u64

Traits§

BoxedSliceEncodable: Define this trait for T to get impl Box<[T]> for CompactEncoding
CompactEncoding: A trait for building small and fast parsers and serializers.
FixedWidthEncoding: Implents functionality needed to encode unisegned integrer in a fixed width way with CompactEncoding
VecEncodable: Implement this for type T to have CompactEncoding implemented for Vec<T>

Functions§

as_array: Get a slice as an array of size N. Errors when slice.len() != N.
as_array_mut: Get a slice as a mutable array of size N. Errors when slice.len() != N.
bytes_fixed_from_vec: Helper to convert a vec to an array, and fail with an encoding error when needed
decode_bytes_fixed: Decode a fixed sized array from a buffer. Return the array and the remainder of the buffer. Errors when buffer.len() < N;
decode_usize: Decode a usize from buffer and return the remaining bytes. This will fail, when we are decoding a usize on a usize = u32 machine for data that was originally encoded on a usize = u64 machine whenever the value is over u32::MAX.
encode_bytes_fixed: Encoded a fixed sized array to a buffer, return the remainder of the buffer. Errors when value.len() > buffer.len();
encode_var_u64: Write uint to the start of buffer and return the remaining part of buffer.
encoded_size_usize: The number of bytes required to encode this number. Note this is always variable width.
encoded_size_var_u64: The number of bytes required to encode this number. We only need this for u64 because all other uints can be converted to usize reliably.
get_slices_checked: Split a slice in two at mid. Returns encoding error when mid is out of bounds.
get_slices_mut_checked: Split a mutable slice into two mutable slices around mid. Returns encoding error when mid is out of bounds.
take_array: split the first N bytes of buffer off and return them
take_array_mut: split the first N bytes of buffer off and return them
write_array: Write source to buffer and return the remainder of buffer. Errors when N < buffer.len()
write_slice: write source to buffer and return remaining buffer

Type Aliases§

FixedWidthU32: A wrapper around u32 to let us encoded/decode to/from a fixed width
FixedWidthU64: A wrapper around u64 to let us encoded/decode to/from a fixed width