Crate deku

Expand description

§Deku: Declarative binary reading and writing

Deriving a struct or enum with DekuRead and DekuWrite provides bit-level, symmetric, serialization/deserialization implementations.

This allows the developer to focus on building and maintaining how the data is represented and manipulated and not on redundant, error-prone, parsing/writing code. This approach is especially useful when dealing with binary structures such as TLVs or network protocols. This allows the internal rustc compiler to choose the in-memory representation of the struct, while reading and writing can understand the struct in a “packed” C way.

Under the hood, many specializations are done in order to achieve performant code. For reading and writing bytes, the std library is used. When bit-level control is required, it makes use of the bitvec crate as the “Reader” and “Writer”.

For documentation and examples on available #[deku] attributes and features, see attributes list

For more examples, see the examples folder!

§no_std

For use in no_std environments, alloc is the single feature which is required on deku.

§Example

Let’s read big-endian data into a struct, with fields containing different sizes, modify a value, and write it back. In this example we use from_bytes, but we could also use from_reader.

use deku::prelude::*;

#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
#[deku(endian = "big")]
struct DekuTest {
    #[deku(bits = 4)]
    field_a: u8,
    #[deku(bits = 4)]
    field_b: u8,
    field_c: u16,
}

let data: Vec<u8> = vec![0b0110_1001, 0xBE, 0xEF];
let (_rest, mut val) = DekuTest::from_bytes((data.as_ref(), 0)).unwrap();
assert_eq!(DekuTest {
    field_a: 0b0110,
    field_b: 0b1001,
    field_c: 0xBEEF,
}, val);

val.field_c = 0xC0FE;

let data_out = val.to_bytes().unwrap();
assert_eq!(vec![0b0110_1001, 0xC0, 0xFE], data_out);

§Composing

Deku structs/enums can be composed as long as they implement DekuReader / DekuWrite traits which can be derived by using the DekuRead and DekuWrite Derive macros.

use deku::prelude::*;

#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
#[deku(endian = "big")]
struct DekuTest {
    header: DekuHeader,
    data: DekuData,
}

#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
#[deku(ctx = "endian: deku::ctx::Endian")] // context passed from `DekuTest` top-level endian
struct DekuHeader(u8);

#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
#[deku(ctx = "endian: deku::ctx::Endian")] // context passed from `DekuTest` top-level endian
struct DekuData(u16);

let data: Vec<u8> = vec![0xAA, 0xEF, 0xBE];
let (_rest, mut val) = DekuTest::from_bytes((data.as_ref(), 0)).unwrap();
assert_eq!(DekuTest {
    header: DekuHeader(0xAA),
    data: DekuData(0xBEEF),
}, val);

let data_out = val.to_bytes().unwrap();
assert_eq!(data, data_out);

Note that because we explicitly specify the endian on the top-level struct, we must pass the endian to all children via the context. Several attributes trigger this requirement, such as endian, bit_order, and bits. If you are getting errors of the format mismatched types, expected A, found B on the DekuRead and/or DekuWrite #[derive] attribute, then you need to update the ctx.

§Vec

Vec can be used in combination with the count attribute (T must implement DekuRead/DekuWrite)

bytes_read or bits_read can also be used instead of count to read a specific size of each.

If the length of Vec changes, the original field specified in count will not get updated. Calling .update() can be used to “update” the field!

use deku::prelude::*;

#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
struct DekuTest {
    #[deku(update = "self.data.len()")]
    count: u8,
    #[deku(count = "count")]
    data: Vec<u8>,
}

let data: Vec<u8> = vec![0x02, 0xBE, 0xEF, 0xFF, 0xFF];
let (_rest, mut val) = DekuTest::from_bytes((data.as_ref(), 0)).unwrap();
assert_eq!(DekuTest {
    count: 0x02,
    data: vec![0xBE, 0xEF]
}, val);

let data_out = val.to_bytes().unwrap();
assert_eq!(vec![0x02, 0xBE, 0xEF], data_out);

// Pushing an element to data
val.data.push(0xAA);

assert_eq!(DekuTest {
    count: 0x02, // Note: this value has not changed
    data: vec![0xBE, 0xEF, 0xAA]
}, val);

let data_out = val.to_bytes().unwrap();
// Note: `count` is still 0x02 while 3 bytes got written
assert_eq!(vec![0x02, 0xBE, 0xEF, 0xAA], data_out);

// Use `update` to update `count`
val.update().unwrap();

assert_eq!(DekuTest {
    count: 0x03,
    data: vec![0xBE, 0xEF, 0xAA]
}, val);

§Enums

As enums can have multiple variants, each variant must have a way to match on the incoming data.

First the “type” is read using id_type, then is matched against the variants given id. What happens after is the same as structs!

This is implemented with the id, id_pat, default and id_type attributes. See these for more examples.

If no id is specified, the variant will default to it’s discriminant value.

If no variant can be matched and the default is not provided, a DekuError::Parse error will be returned.

If no variant can be matched and the default is provided, a variant will be returned based on the field marked with default.

Example:

use deku::prelude::*;

#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
#[deku(id_type = "u8")]
enum DekuTest {
    #[deku(id = 0x01)]
    VariantA,
    #[deku(id = 0x02)]
    VariantB(u16),
}

let data: &[u8] = &[0x01, 0x02, 0xEF, 0xBE];
let mut cursor = Cursor::new(data);

let (_, val) = DekuTest::from_reader((&mut cursor, 0)).unwrap();
assert_eq!(DekuTest::VariantA , val);

// cursor now points at 0x02
let (_, val) = DekuTest::from_reader((&mut cursor, 0)).unwrap();
assert_eq!(DekuTest::VariantB(0xBEEF) , val);

Of course, trivial c-style enums works just as well too:


#[derive(Debug, PartialEq, DekuRead, DekuWrite)]
#[deku(id_type = "u8", bits = 2, bit_order = "lsb")]
#[repr(u8)]
pub enum DekuTest {
    VariantA = 0,
    VariantB = 1,
    VariantC = 2,
    VariantD = 3
}

let data: &[u8] = &[0x0D]; // 00 00 11 01 => A A D B
let mut cursor = Cursor::new(data);
let mut reader = Reader::new(&mut cursor);

let val = DekuTest::from_reader_with_ctx(&mut reader, ()).unwrap();
assert_eq!(DekuTest::VariantB , val);

let val = DekuTest::from_reader_with_ctx(&mut reader, ()).unwrap();
assert_eq!(DekuTest::VariantD , val);

let val = DekuTest::from_reader_with_ctx(&mut reader, ()).unwrap();
assert_eq!(DekuTest::VariantA , val);

§Context

Child parsers can get access to the parent’s parsed values using the ctx attribute

For more information see ctx attribute

Example:

use deku::prelude::*;

#[derive(DekuRead, DekuWrite)]
#[deku(ctx = "a: u8")]
struct Subtype {
    #[deku(map = "|b: u8| -> Result<_, DekuError> { Ok(b + a) }")]
    b: u8
}

#[derive(DekuRead, DekuWrite)]
struct Root {
    a: u8,
    #[deku(ctx = "*a")] // `a` is a reference
    sub: Subtype
}

let data: &[u8] = &[0x01, 0x02];
let mut cursor = Cursor::new(data);

let (amt_read, value) = Root::from_reader((&mut cursor, 0)).unwrap();
assert_eq!(value.a, 0x01);
assert_eq!(value.sub.b, 0x01 + 0x02)

§`Read` supported

Parsers can be created that directly read from a source implementing Read.

The crate no_std_io2 is re-exported as no_std_io for use in no_std environments. This functions as an alias for std::io when not using no_std.

#[derive(Debug, DekuRead, DekuWrite, PartialEq, Eq, Clone)]
#[deku(endian = "big")]
struct EcHdr {
    magic: [u8; 4],
    version: u8,
    padding1: [u8; 3],
}

let mut file = File::options().read(true).open("file").unwrap();
let ec = EcHdr::from_reader((&mut file, 0)).unwrap();

§`Write` supported

Parsers can be created that directly write to a source implementing Write.

#[derive(Debug, DekuRead, DekuWrite, PartialEq, Eq, Clone)]
#[deku(endian = "big")]
struct Hdr {
    version: u8,
}

let hdr = Hdr { version: 0xf0 };
let mut file = File::options().write(true).open("file").unwrap();
hdr.to_writer(&mut Writer::new(file), ());

§DekuSize

For types with a known, fixed size at compile-time, the DekuSize trait provides constant SIZE_BITS and SIZE_BYTES values. This is useful for creating correctly sized buffers in embedded or no_std,no_alloc environments where dynamic allocation is not available.

use deku::prelude::*;

#[derive(DekuRead, DekuWrite, DekuSize)]
#[deku(endian = "big")]
struct Message {
    msg_type: u8,
    payload: [u8; 16],
    checksum: u16,
}

assert_eq!(Message::SIZE_BYTES, Some(19));

const BUFFER_SIZE: usize = Message::SIZE_BYTES.unwrap();
let mut buffer = [0u8; BUFFER_SIZE];

let msg = Message {
    msg_type: 0x01,
    payload: [0xFF; 16],
    checksum: 0xABCD,
};

let written = msg.to_slice(&mut buffer).unwrap();
assert_eq!(written, BUFFER_SIZE);

For enums, SIZE_BITS represents the discriminant plus the maximum variant size:

use deku::prelude::*;

#[derive(DekuRead, DekuWrite, DekuSize)]
#[deku(id_type = "u8")]
enum Packet {
    #[deku(id = "1")]
    Small { data: u16 },
    #[deku(id = "2")]
    Large { data: u64 },
}

assert_eq!(Packet::SIZE_BYTES, Some(9));

const MAX_SIZE: usize = Packet::SIZE_BYTES.unwrap();
let mut buffer = [0u8; MAX_SIZE];

Note: Variable-size types like Vec do not implement DekuSize as their size cannot be known at compile-time.

§Internal variables and previously read fields

Along similar lines to Context variables, previously read variables are exposed and can be referenced:

Example:

#[derive(DekuRead)]
struct DekuTest {
    num_items: u8,
    #[deku(count = "num_items")]
    items: Vec<u16>,
}

The following variables are internals which can be used in attributes accepting tokens such as reader, writer, map, count, etc.

These are provided as a convenience to the user.

Always included:

deku::reader: &mut Reader - Current Reader
deku::writer: &mut Writer - Current Writer

Conditionally included if referenced:

deku::bit_offset: usize - Current bit offset from the input
deku::byte_offset: usize - Current byte offset from the input

Example:

#[derive(DekuRead)]
#[deku(ctx = "size: u32")]
pub struct EncodedString {
    encoding: u8,

    #[deku(count = "size as usize - deku::byte_offset")]
    data: Vec<u8>
}

§Debugging decoders with the `logging` feature.

If you are having trouble understanding what causes a Deku parse error, you may find the logging feature useful.

To use it, you will need to:

enable the logging Cargo feature for your Deku dependency
import the log crate and a compatible logging library

For example, to log with env_logger, the dependencies in your Cargo.toml might look like:

deku = { version = "*", features = ["logging"] }
log = "*"
env_logger = "*"

Then you’d call env_logger::init() or env_logger::try_init() prior to doing Deku decoding.

Deku uses the trace logging level, so if you run your application with RUST_LOG=trace in your environment, you will see logging messages as Deku does its deserialising.

§Reducing parser code size

With the use of the no-assert-string feature, you can remove the strings Deku adds to assertion errors.
DekuError whenever possible will use a 'static str, to make the errors compile away when following a guide such as min-sized-rust.

§Performance: Compile without `bitvec`

The feature bits enables the bitvec crate to use when reading and writing, which is enabled by default. This however slows down the reading and writing process if your code doesn’t use bits and the bit_offset in from_bytes.

§NoSeek

Unseekable streams such as TcpStream are supported through the NoSeek wrapper.

Re-exports§

pub use crate::error::DekuError;

Modules§

attributes: A documentation-only module for #[deku] attributes
bitvec: re-export of bitvec
ctx: Types for context representation See ctx attribute for more information.
error: Error module
no_std_io: re-export of no_std_io2
noseek: Wrapper type that provides a fake Seek implementation.
prelude: Crate prelude
reader: Reader for reader functions
writer: Writer for writer functions

Macros§

deku_error: Abstract over alloc vs no-alloc for handling of error strings

Structs§

BoundedBitVec: Like BitVec but with bounded, local storage

Traits§

DekuContainerRead: “Reader” trait: implemented on DekuRead struct and enum containers. A container is a type which doesn’t need any context information.
DekuContainerWrite: “Writer” trait: implemented on DekuWrite struct and enum containers. A container is a type which doesn’t need any context information.
DekuEnumExt: “Extended Enum” trait: obtain additional enum information
DekuReader: “Reader” trait: read bytes and bits from no_std_io::Reader
DekuSize: Trait for types with a known, fixed binary size at compile-time
DekuUpdate: “Updater” trait: apply mutations to a type
DekuWriter: “Writer” trait: write from type to bytes

Attribute Macros§

deku_derive: Entry function for deku_derive proc-macro This attribute macro is used to derive DekuRead and DekuWrite while removing temporary variables.

Derive Macros§

DekuRead: Entry function for DekuRead proc-macro
DekuSize: Entry function for DekuSize proc-macro
DekuWrite: Entry function for DekuWrite proc-macro

Crate deku

Crate deku Copy item path

§Deku: Declarative binary reading and writing

§no_std

§Example

§Composing

§Vec

§Enums

§Context

§Read supported

§Write supported

§DekuSize

§Internal variables and previously read fields

§Debugging decoders with the logging feature.

§Reducing parser code size

§Performance: Compile without bitvec

§NoSeek

Re-exports§

Modules§

Macros§

Structs§

Traits§

Attribute Macros§

Derive Macros§

Crate deku

§`Read` supported

§`Write` supported

§Debugging decoders with the `logging` feature.

§Performance: Compile without `bitvec`