Crate parsely_rs

Source
Expand description

§Parsely

Convenient type serialization and deserialization in Rust for binary formats.

Parsely uses derive macros to automatically implement serialization and deserialization methods for your types.

This crate is heavily inspired by the Deku crate (and is nowhere near as complete). See Differences from Deku below.

§Example

Say you want to parse an RTCP header formatted like so:

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|    SC   |      PT       |             length            |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

Where the version (V) field should always contain the value 2. The code to serialize and deserialize it can be written with Parsely like this:


#[derive(Debug, PartialEq, Eq, ParselyRead, ParselyWrite)]
pub struct RtcpHeader {
    #[parsely(assertion = "|v: &u2| *v == 2")]
    pub version: u2,
    pub has_padding: bool,
    pub report_count: u5,
    pub packet_type: u8,
    pub length_field: u16,
}

// Reading it from a buffer
fn do_read(data: Vec<u8>) {
  let mut bits = Bits::from_owner_bytes(data);

  let rtcp_header = RtcpHeader::read::<NetworkOrder>(&mut bits, ())
    .context("Reading RtcpHeader")
    .unwrap();
}

// Writing it out to a buffer
fn do_write(rtcp_header: RtcpHeader) {
  let mut bits_mut = BitsMut::new();
  
  let result = rtcp_header.write::<NetworkOrder>(&mut bits_mut, ());
}

§Traits

The ParselyRead trait is used for reading data from a buffer. ParselyRead can be derived and its logic customized via the attributes described below, but can also be manually implemented.


pub trait ParselyRead<B>: Sized {
    type Ctx;
    fn read<T: ByteOrder>(buf: &mut B, ctx: Self::Ctx) -> ParselyResult<Self>;
}

The ParselyWrite trait is used for writing data to a buffer. Like ParselyRead, ParselyWrite can be derived and customized or manually implemented.


pub trait ParselyWrite<B>: StateSync + Sized {
    type Ctx;
    fn write<T: ByteOrder>(&self, buf: &mut B, ctx: Self::Ctx) -> ParselyResult<()>;
}

The StateSync trait is a required supertrait of ParselyWrite and enforces synchronization of fields before writing.

use parsely_rs::*;

pub trait StateSync: Sized {
    type SyncCtx;

    fn sync(&mut self, sync_ctx: Self::SyncCtx) -> ParselyResult<()> {
        Ok(())
    }
}

When deriving ParselyWrite, a StateSync implementation will be generated as well. See the dependent fields section for more information on how attributes can be used to customize the behavior. If you manually implement ParselyWrite yourself, you’ll need to implement StateSync as well. If the field requires no synchronization, you can use the impl_stateless_sync macro to generate a default impl for your type.

Sometimes serializing or deserializing a type requires additional data that may come from somewhere else. The Ctx generic can be defined as a tuple and the ctx argument can be used to pass additional values.

See the Context and required context section below for more information.

The ByteOrder generic is used to describe how the data is laid out in the buffer (e.g. LittleEndian or BigEndian). The B generic is the buffer type. Typically this is an instance of BitBuf for reading and BitBufMut for writing. Both types come from the bit-cursor crate.

§Attributes

Parsely defines various attributes to make parsing different structures possible. There are 3 modes of applying attributes:

  • read + write via #[parsely]
  • read-only via #[parsely_read]
  • write-only via #[parsely_write]

Some attributes are only available for reading or writing

§Assertion

An assertion is applied to the value pulled from the buffer after reading or to the field before writing. They allow reading and/or writing to fail when the assertion fails. An assertion can either be a closure or the path to a function. Both styles must be functions which take a reference to the value’s type and return a boolean.

ModeAvailable
#[parsely]:white_check_mark:
#[parsely_read]:white_check_mark:
#[parsely_write]:white_check_mark:
§Examples
Click to expand
use parsely_rs::*;

#[derive(Debug, ParselyRead, ParselyWrite)]
pub struct MyStruct {
  #[parsely(assertion = "|v: &u8| *v == 42")]
  pub value: u8,
}
use parsely_rs::*;

fn my_assertion(value: &u8) -> bool {
  *value == 42
}

#[derive(Debug, ParselyRead, ParselyWrite)]
pub struct MyStruct {
  #[parsely(assertion = "my_assertion")]
  pub value: u8,
}

§Map

A transformation may be applied to a value read from a buffer before assigning it to the field, or to a field’s value before writing it to the buffer.

Because the signatures for read and write map functions are slightly different, the map attribute must be applied independently for reading and writing via #[parsely_read] and #[parsely_write]

When passed via #[parsely_read], the argument must evaluate to a function or a closure which takes a type T by value where T: ParselyRead and can return either a type U or a Result<U, E> where U is the type of the field and E: Into<anyhow::Error>.

When passed via #[parsely_write], the argument must evaluate to a function or closure which takes a reference to a type T, where T is the type of the field and returns either a type U or a Result<U, E> where U: ParselyWrite and E: Into<anyhow::Error>.

ModeAvailable
#[parsely]:x:
#[parsely_read]:white_check_mark:
#[parsely_write]:white_check_mark:
§Examples
Click to expand

This example has a String field but reads a u8 from the buffer and converts it. On write it does the opposite.

use parsely_rs::*;

#[derive(ParselyRead, ParselyWrite)]
struct Foo {
    // Closures can return a raw value...
    #[parsely_read(map = "|v: u8| { v.to_string() }")]
    // ...or a Result<T, E> as long as E: Into<anyhow::Error>
    #[parsely_write(map = "|v: &str| { v.parse::<u8>() }")]
    value: String,
}

let mut bits = Bits::from_static_bytes(&[42]);

let foo = Foo::read::<NetworkOrder>(&mut bits, ()).expect("successful read");
assert_eq!(foo.value, "42");

let mut bits_mut = BitsMut::new();
foo.write::<NetworkOrder>(&mut bits_mut, ()).expect("successful write");
assert_eq!(bits_mut.freeze(), Bits::from_static_bytes(&[42]));

§Count

When reading a Vec<T>, we need to know how many elements to read. The count attribute is used to describe how many elements should be read from the buffer.

Any expression that evaluates to a number that can be used in a range expression can be used.

ModeAvailable
#[parsely]:x:
#[parsely_read]:white_check_mark:
#[parsely_write]:x:
§Examples
Click to expand

Here a u8 is read into the data_size field and the value of that field is used to denote the number of elements.

use parsely_rs::*;

#[derive(ParselyRead, ParselyWrite)]
struct Foo {
    data_size: u8,
    // Here we refer to the previously-read 'data_size' field to describe the length
    #[parsely_read(count = "data_size")]
    data: Vec<u8>,
}

§When

Optional fields need to be given a predicate that describe when they should be attempted to be read. The when attribute takes an expression that evaluates to a boolean. A result of true means the field will be read from the buffer, false means it will be skipped and set to None.

ModeAvailable
#[parsely]:x:
#[parsely_read]:white_check_mark:
#[parsely_write]:x:
§Examples
Click to expand

Here, a boolean value is read into the has_value field and whether a u32 is read for value field is based on if has_value is true or false.

use parsely_rs::*;

#[derive(ParselyRead, ParselyWrite)]
struct Foo {
    has_value: bool,
    // Here we refer to the previously-read 'has_value' field to 
    // describe whether or not this field is present
    #[parsely_read(when = "has_value")]
    value: Option<u32>,
}

§Assign from

Sometimes a field should be assigned to a value rather than read from the buffer. Any expression evaluating to the type of the field can be passed.

ModeAvailable
#[parsely]:x:
#[parsely_read]:white_check_mark:
#[parsely_write]:x:
§Examples
Click to expand

Here the header value has already been read and is passed in via context. It is then assigned directly to the header field.

use parsely_rs::*;

#[derive(ParselyRead, ParselyWrite)]
struct Header {
  payload_type: u8,
}

#[derive(ParselyRead, ParselyWrite)]
#[parsely_read(required_context("header: Header"))]
struct Packet {
  #[parsely_read(assign_from = "header")]
  header: Header,
  other_field: u8,
}

§Dependent fields

Often times packets will have fields whose values depend on other fields. A header might have a length field that should reflect the size of a payload. Parsely defines multiple attributes to define these relationships:

The sync_args attribute is used on a struct to define what external information is needed in order to sync its fields correctly.

The sync_expr attribute is used on a specific field to define how it should use the values from sync_args (or elsewhere) in order to sync.

The sync_with attribute is used to pass information to a field to synchronize it.

All types that implement ParselyWrite must also implement the StateSync trait.

The sync function from the StateSync trait should be called explicitly before writing the type to a buffer to make sure all fields are consistent.

ModeAvailable
#[parsely]:x:
#[parsely_read]:x:
#[parsely_write]:white_check_mark:
§Examples
Click to expand

Here, a header contains a length field that should describe the length of the entire packet. The payload contains a variable-length array, so its length needs to be taken into account rest of the payload. A field from the header is passed as context to the payload parsing.

use parsely_rs::*;

#[derive(Debug, ParselyWrite)]
// sync_args denotes that this type's sync method takes additional 
// arguments.  By default a type's sync field takes no arguments
#[parsely_write(sync_args("payload_length_bytes: u16"))]
struct Header {
    version: u8,
    packet_type: u8,
    // sync_func can refer to an expression or a function and will be used to
    // update the annotated // field, it should evaluate to ParselyResult<T> 
    // where T is the type of the field.  You can // refer to variables defined in
    // sync_args.
    #[parsely_write(sync_expr = "ParselyResult::Ok(payload_length_bytes + 4)")]
    length_bytes: u16,
}

#[derive(Debug, ParselyWrite)]
struct Packet {
    // sync_with attributes add lines to this type's sync method to call 
    // sync on its fields (and what arguments to pass)
    #[parsely_write(sync_with("self.data.len() as u16"))]
    header: Header,
    data: Vec<u8>,
}

let mut packet = Packet {
    header: Header {
        version: 1,
        packet_type: 2,
        length_bytes: 0,
    },
    data: vec![1, 2, 3, 4],
};

packet.sync(()).unwrap();

assert_eq!(packet.header.length_bytes, 8);

§Context and required context

Sometimes in order to read or write a struct or field, additional data is needed. Structs can declare what additional data is needed via the required_context attribute. Additional data can be also be passed down to fields via the context attribute. Any required_context or previously-parsed field name can be used.

The argument passed to required_context is a comma-separated list of typed function arguments (e.g. size: u8, name: String). The variable names there can be used in other attributes.

The argument passed to context is a comma-separated list of expressions that evaluate to values that should be passed to that field’s read and/or write method.

ModeAvailable
#[parsely]:white_check_mark:
#[parsely_read]:white_check_mark:
#[parsely_write]:white_check_mark:
§Examples
Click to expand

Here, a header is parsed first which contains information needed to parse the rest of the payload. A field from the header is passed as context to the payload parsing.

use parsely_rs::*;

#[derive(ParselyRead, ParselyWrite)]
struct FooHeader {
  packet_type: u8,
  payload_len: u32,
}

#[derive(ParselyRead, ParselyWrite)]
// Foo needs additional context in order to be parsed from a buffer
#[parsely_read(required_context("len: u32"))]
struct Foo {
    // The required_context variable is accessible and can be referred to when
    // describing the length of the Vec that should be read
    #[parsely_read(count = "len")]
    data: Vec<u8>,
}

fn run(buf: &mut Bits) {
  let foo_header = FooHeader::read::<NetworkOrder>(buf, ()).unwrap();
  // Pass the relevant field from header to the payload's read method
  let foo_payload = Foo::read::<NetworkOrder>(buf, (foo_header.payload_len,)).unwrap();
}

§TODO/Roadmap

  • Probably need some more options around collections (e.g. while)

§Differences from Deku

The original intent for writing this crate was to come up with a straightforward, generic way to quickly write serialization and deserialization code for packets. It does not strive to be a “better Deku”: if you’re writing any sort of production code, Deku is what you want. The goal here was to have an excuse to play around with derive macros and have a library that I could leverage for other personal projects. That being said, here are a couple decisions I made that, from what I can tell, are different from Deku:

  1. The nsw-types crate is used to describe fields of non-standard widths (u3, u18, u33, etc. as opposed to using u8, u16, etc. and specifying the number of bits via an attribute), which makes message definitions more explicitly-typed and eliminates the needs for extra attributes. The tradeoff here is that a special cursor type (BitCursor) is required to process the buffer.

  2. Byte order is specified as part of the read and write calls as opposed to the struct definition. Deku may support this as well, but I didn’t even add attributes to denote a type’s byte order because it felt like that should exist outside the type’s definition.

  3. More…

Modules§

from_bitslice
trait_impls

Macros§

anyhow
Construct an ad-hoc error from a string or existing non-anyhow error value.
bail
Return early with an error.
impl_stateless_sync

Structs§

BigEndian
BitCursor
Bits
A cheaply cloneable chunk of contiugous memory, built on top of [bytes::Bytes] but providing bit-level operations.
BitsMut
LittleEndian
i1
The 1-bit signed integer type.
i2
The 2-bit signed integer type.
i3
The 3-bit signed integer type.
i4
The 4-bit signed integer type.
i5
The 5-bit signed integer type.
i6
The 6-bit signed integer type.
i7
The 7-bit signed integer type.
i9
The 9-bit signed integer type.
i10
The 10-bit signed integer type.
i11
The 11-bit signed integer type.
i12
The 12-bit signed integer type.
i13
The 13-bit signed integer type.
i14
The 14-bit signed integer type.
i15
The 15-bit signed integer type.
i17
The 17-bit signed integer type.
i18
The 18-bit signed integer type.
i19
The 19-bit signed integer type.
i20
The 20-bit signed integer type.
i21
The 21-bit signed integer type.
i22
The 22-bit signed integer type.
i23
The 23-bit signed integer type.
i24
The 24-bit signed integer type.
i25
The 25-bit signed integer type.
i26
The 26-bit signed integer type.
i27
The 27-bit signed integer type.
i28
The 28-bit signed integer type.
i29
The 29-bit signed integer type.
i30
The 30-bit signed integer type.
i31
The 31-bit signed integer type.
i33
The 33-bit signed integer type.
i34
The 34-bit signed integer type.
i35
The 35-bit signed integer type.
i36
The 36-bit signed integer type.
i37
The 37-bit signed integer type.
i38
The 38-bit signed integer type.
i39
The 39-bit signed integer type.
i40
The 40-bit signed integer type.
i41
The 41-bit signed integer type.
i42
The 42-bit signed integer type.
i43
The 43-bit signed integer type.
i44
The 44-bit signed integer type.
i45
The 45-bit signed integer type.
i46
The 46-bit signed integer type.
i47
The 47-bit signed integer type.
i48
The 48-bit signed integer type.
i49
The 49-bit signed integer type.
i50
The 50-bit signed integer type.
i51
The 51-bit signed integer type.
i52
The 52-bit signed integer type.
i53
The 53-bit signed integer type.
i54
The 54-bit signed integer type.
i55
The 55-bit signed integer type.
i56
The 56-bit signed integer type.
i57
The 57-bit signed integer type.
i58
The 58-bit signed integer type.
i59
The 59-bit signed integer type.
i60
The 60-bit signed integer type.
i61
The 61-bit signed integer type.
i62
The 62-bit signed integer type.
i63
The 63-bit signed integer type.
i65
The 65-bit signed integer type.
i66
The 66-bit signed integer type.
i67
The 67-bit signed integer type.
i68
The 68-bit signed integer type.
i69
The 69-bit signed integer type.
i70
The 70-bit signed integer type.
i71
The 71-bit signed integer type.
i72
The 72-bit signed integer type.
i73
The 73-bit signed integer type.
i74
The 74-bit signed integer type.
i75
The 75-bit signed integer type.
i76
The 76-bit signed integer type.
i77
The 77-bit signed integer type.
i78
The 78-bit signed integer type.
i79
The 79-bit signed integer type.
i80
The 80-bit signed integer type.
i81
The 81-bit signed integer type.
i82
The 82-bit signed integer type.
i83
The 83-bit signed integer type.
i84
The 84-bit signed integer type.
i85
The 85-bit signed integer type.
i86
The 86-bit signed integer type.
i87
The 87-bit signed integer type.
i88
The 88-bit signed integer type.
i89
The 89-bit signed integer type.
i90
The 90-bit signed integer type.
i91
The 91-bit signed integer type.
i92
The 92-bit signed integer type.
i93
The 93-bit signed integer type.
i94
The 94-bit signed integer type.
i95
The 95-bit signed integer type.
i96
The 96-bit signed integer type.
i97
The 97-bit signed integer type.
i98
The 98-bit signed integer type.
i99
The 99-bit signed integer type.
i100
The 100-bit signed integer type.
i101
The 101-bit signed integer type.
i102
The 102-bit signed integer type.
i103
The 103-bit signed integer type.
i104
The 104-bit signed integer type.
i105
The 105-bit signed integer type.
i106
The 106-bit signed integer type.
i107
The 107-bit signed integer type.
i108
The 108-bit signed integer type.
i109
The 109-bit signed integer type.
i110
The 110-bit signed integer type.
i111
The 111-bit signed integer type.
i112
The 112-bit signed integer type.
i113
The 113-bit signed integer type.
i114
The 114-bit signed integer type.
i115
The 115-bit signed integer type.
i116
The 116-bit signed integer type.
i117
The 117-bit signed integer type.
i118
The 118-bit signed integer type.
i119
The 119-bit signed integer type.
i120
The 120-bit signed integer type.
i121
The 121-bit signed integer type.
i122
The 122-bit signed integer type.
i123
The 123-bit signed integer type.
i124
The 124-bit signed integer type.
i125
The 125-bit signed integer type.
i126
The 126-bit signed integer type.
i127
The 127-bit signed integer type.
u1
The 1-bit unsigned integer type.
u2
The 2-bit unsigned integer type.
u3
The 3-bit unsigned integer type.
u4
The 4-bit unsigned integer type.
u5
The 5-bit unsigned integer type.
u6
The 6-bit unsigned integer type.
u7
The 7-bit unsigned integer type.
u9
The 9-bit unsigned integer type.
u10
The 10-bit unsigned integer type.
u11
The 11-bit unsigned integer type.
u12
The 12-bit unsigned integer type.
u13
The 13-bit unsigned integer type.
u14
The 14-bit unsigned integer type.
u15
The 15-bit unsigned integer type.
u17
The 17-bit unsigned integer type.
u18
The 18-bit unsigned integer type.
u19
The 19-bit unsigned integer type.
u20
The 20-bit unsigned integer type.
u21
The 21-bit unsigned integer type.
u22
The 22-bit unsigned integer type.
u23
The 23-bit unsigned integer type.
u24
The 24-bit unsigned integer type.
u25
The 25-bit unsigned integer type.
u26
The 26-bit unsigned integer type.
u27
The 27-bit unsigned integer type.
u28
The 28-bit unsigned integer type.
u29
The 29-bit unsigned integer type.
u30
The 30-bit unsigned integer type.
u31
The 31-bit unsigned integer type.
u33
The 33-bit unsigned integer type.
u34
The 34-bit unsigned integer type.
u35
The 35-bit unsigned integer type.
u36
The 36-bit unsigned integer type.
u37
The 37-bit unsigned integer type.
u38
The 38-bit unsigned integer type.
u39
The 39-bit unsigned integer type.
u40
The 40-bit unsigned integer type.
u41
The 41-bit unsigned integer type.
u42
The 42-bit unsigned integer type.
u43
The 43-bit unsigned integer type.
u44
The 44-bit unsigned integer type.
u45
The 45-bit unsigned integer type.
u46
The 46-bit unsigned integer type.
u47
The 47-bit unsigned integer type.
u48
The 48-bit unsigned integer type.
u49
The 49-bit unsigned integer type.
u50
The 50-bit unsigned integer type.
u51
The 51-bit unsigned integer type.
u52
The 52-bit unsigned integer type.
u53
The 53-bit unsigned integer type.
u54
The 54-bit unsigned integer type.
u55
The 55-bit unsigned integer type.
u56
The 56-bit unsigned integer type.
u57
The 57-bit unsigned integer type.
u58
The 58-bit unsigned integer type.
u59
The 59-bit unsigned integer type.
u60
The 60-bit unsigned integer type.
u61
The 61-bit unsigned integer type.
u62
The 62-bit unsigned integer type.
u63
The 63-bit unsigned integer type.
u65
The 65-bit unsigned integer type.
u66
The 66-bit unsigned integer type.
u67
The 67-bit unsigned integer type.
u68
The 68-bit unsigned integer type.
u69
The 69-bit unsigned integer type.
u70
The 70-bit unsigned integer type.
u71
The 71-bit unsigned integer type.
u72
The 72-bit unsigned integer type.
u73
The 73-bit unsigned integer type.
u74
The 74-bit unsigned integer type.
u75
The 75-bit unsigned integer type.
u76
The 76-bit unsigned integer type.
u77
The 77-bit unsigned integer type.
u78
The 78-bit unsigned integer type.
u79
The 79-bit unsigned integer type.
u80
The 80-bit unsigned integer type.
u81
The 81-bit unsigned integer type.
u82
The 82-bit unsigned integer type.
u83
The 83-bit unsigned integer type.
u84
The 84-bit unsigned integer type.
u85
The 85-bit unsigned integer type.
u86
The 86-bit unsigned integer type.
u87
The 87-bit unsigned integer type.
u88
The 88-bit unsigned integer type.
u89
The 89-bit unsigned integer type.
u90
The 90-bit unsigned integer type.
u91
The 91-bit unsigned integer type.
u92
The 92-bit unsigned integer type.
u93
The 93-bit unsigned integer type.
u94
The 94-bit unsigned integer type.
u95
The 95-bit unsigned integer type.
u96
The 96-bit unsigned integer type.
u97
The 97-bit unsigned integer type.
u98
The 98-bit unsigned integer type.
u99
The 99-bit unsigned integer type.
u100
The 100-bit unsigned integer type.
u101
The 101-bit unsigned integer type.
u102
The 102-bit unsigned integer type.
u103
The 103-bit unsigned integer type.
u104
The 104-bit unsigned integer type.
u105
The 105-bit unsigned integer type.
u106
The 106-bit unsigned integer type.
u107
The 107-bit unsigned integer type.
u108
The 108-bit unsigned integer type.
u109
The 109-bit unsigned integer type.
u110
The 110-bit unsigned integer type.
u111
The 111-bit unsigned integer type.
u112
The 112-bit unsigned integer type.
u113
The 113-bit unsigned integer type.
u114
The 114-bit unsigned integer type.
u115
The 115-bit unsigned integer type.
u116
The 116-bit unsigned integer type.
u117
The 117-bit unsigned integer type.
u118
The 118-bit unsigned integer type.
u119
The 119-bit unsigned integer type.
u120
The 120-bit unsigned integer type.
u121
The 121-bit unsigned integer type.
u122
The 122-bit unsigned integer type.
u123
The 123-bit unsigned integer type.
u124
The 124-bit unsigned integer type.
u125
The 125-bit unsigned integer type.
u126
The 126-bit unsigned integer type.
u127
The 127-bit unsigned integer type.

Traits§

BitBuf
BitBufExts
BitBufMut
BitBufMutExts
BitRead
The BitRead trait allows for reading bits from a source.
BitSliceUxExts
BitWrite
A trait for objects which are bit-oriented sinks.
ByteOrder
This trait defines operations to load and store integral values from a buffer, and enables implementing them in different ways for the different byte orders (Big Endian and Little Endian).
Context
Provides the context method for Result.
IntoParselyResult
When we need to convert an expression that may or may not be wrapped in a Result on the read path, we can rely on the fact that we’ll eventually be assigning the value to a field with a concrete type and we can rely on type inference in order to figure out what that should be. Because of that we don’t want/need the ParselyWrite trait bounds on the impl like we have above for the writable side, so we need a different trait here.
IntoWritableParselyResult
Helper trait to coerce values of both T: ParselyWrite and Result<T, E>: E: Into<anyhow::Error> into ParselyResult<T>. We need a trait specifically for writing because if we don’t bound the impl for T in some way there’s ambiguity: the compiler doesn’t know if
ParselyRead
ParselyWrite
StateSync
A trait for syncing a field with any required context. In order to prevent accidental misses of this trait, it’s required for all ParselyWrite implementors. When generating the ParselyWrite implementation, sync will be called on every field.

Type Aliases§

NetworkOrder
ParselyResult

Derive Macros§

ParselyRead
ParselyWrite