Skip to main content

Crate commonware_codec

Crate commonware_codec 

Source
Expand description

Serialize structured data.

§Overview

Provides traits and implementations for efficient and safe binary serialization and deserialization of structured data. The library focuses on:

  • Performance: Uses the bytes crate and aims to minimize allocations.
  • Safety: Deserialization of untrusted data is made safer via the Cfg associated type in the Read trait, allowing users to impose limits (like maximum lengths) or other strict constraints on the data.
  • Ease of Use: Provides implementations for common Rust types and uses extension traits (ReadExt, DecodeExt, etc.) for ergonomic usage.

§Core Concepts

The library revolves around a few core traits:

  • Write: Implement this to define how your type is written to a byte buffer.
  • Read: Implement this to define how your type is read from a byte buffer. It has an associated Cfg type, primarily used to enforce constraints (e.g., size limits) when reading untrusted data. Use () if no config is needed.
  • EncodeSize: Implement this to calculate the exact encoded byte size of a value. Required for efficient buffer pre-allocation.
  • FixedSize: Marker trait for types whose encoded size is constant. Automatically implements EncodeSize and MaxEncodeSize.
  • MaxEncodeSize: Provides an upper bound on encoded size that does not depend on caller-supplied limits.

Helper traits combine these for convenience:

§Size Bounds

Bounds reserve each scalar’s maximum encoded type width, even when validation restricts its values. For example, a UInt<u64> field contributes ten bytes even when its valid values fit in one. A bound therefore need not be attained by any valid value.

Size functions accept caller-supplied limits for variable contents, such as item counts or byte lengths. Length prefixes still contribute their type’s maximum width. For example, util::max_items_size gives a 133-byte bound for a Vec<u8> limited to 128 bytes: 128 bytes of data and a five-byte UInt<u32> length prefix.

FixedSize types contribute their fixed size. Composite types sum their fields’ bounds, and enums add their tag’s bound to the largest variant bound. A size helper belongs with the encoding it describes, so enclosing types can reuse its result.

§Decode Inputs

Readers accept Buf inputs so decoded byte fields can share the input allocation. Pass owned buffers such as ::bytes::Bytes directly, or borrow ::bytes::Bytes through Viewing for an independent cursor without cloning the handle. Both preserve shared storage for retained byte fields. Decoders also accept owned Vec<u8> values through Input and take ownership of their allocation without copying the payload.

Borrowed slices require Copying. Creating this adapter does not allocate, so scalar and byte-array reads can use reusable scratch storage. Fields that retain bytes copy their contents when decoded through the adapter.

Use Buf for generic readers of serialized values and every helper on that read path, including fixed-size reads and length or padding validation. Use Input at entry points that convert owned inputs into readable buffers, then preserve Buf internally.

Use ::bytes::Buf for raw buffer implementations and byte-stream inputs to I/O, encoding, or hashing. When its cursor methods must be in scope alongside Buf, import it as use bytes::Buf as _;.

§Specialization

Byte-oriented container paths use hidden trait hooks on Write, Read, and EncodeSize to select bulk-copy implementations while keeping generic fallbacks. Container implementations call hooks such as T::write_slice, T::read_vec, and T::encode_size_slice. The default methods preserve element-by-element behavior, while concrete element implementations can override only the paths they can make faster.

Encoding specialization has two parts: aggregate sizing and aggregate writing. For example, Vec<u8>::encode() first asks for the output size, then writes the bytes. The EncodeSize slice hooks let fixed-size elements compute SIZE * len without scanning every element, while the Write slice hooks let byte containers write the payload with one bulk copy.

These hooks keep the container code generic: a container like Vec<T> calls one element-level method for sizing, writing, or reading, and the element implementation decides whether the default element-by-element behavior or a bulk path applies.

§Supported Types

Natively supports encoding/decoding for:

With the std feature (enabled by default):

§Implementing for Custom Types

You typically need to implement Write, EncodeSize (unless FixedSize), and Read for your custom structs and enums.

§Example 1. Fixed-Size Type

use bytes::BufMut;
use commonware_codec::{Buf, Error, FixedSize, Read, ReadExt, Write, Encode, DecodeExt};

// Define a custom struct
#[derive(Debug, Clone, PartialEq)]
struct Point {
    x: u32, // FixedSize
    y: u32, // FixedSize
}

// 1. Implement Write: How to serialize the struct
impl Write for Point {
    fn write(&self, buf: &mut impl BufMut) {
        // u32 implements Write
        self.x.write(buf);
        self.y.write(buf);
    }
}

// 2. Implement FixedSize (provides EncodeSize automatically)
impl FixedSize for Point {
    // u32 implements FixedSize
    const SIZE: usize = u32::SIZE + u32::SIZE;
}

// 3. Implement Read: How to deserialize the struct (uses default Cfg = ())
impl Read for Point {
    type Cfg = ();
    fn read_cfg(buf: &mut impl Buf, _: &()) -> Result<Self, Error> {
        // Use ReadExt::read for ergonomic reading when Cfg is ()
        let x = u32::read(buf)?;
        let y = u32::read(buf)?;
        Ok(Self { x, y })
    }
}

// Point now automatically implements Encode, Decode, Codec
let point = Point { x: 1, y: 2 };

// Encode is available via FixedSize + Write
let bytes = point.encode();
assert_eq!(bytes.len(), Point::SIZE);

// Decode is available via Read, use DecodeExt
let decoded_point = Point::decode(bytes).unwrap();
assert_eq!(point, decoded_point);

§Example 2. Variable-Size Type

use bytes::BufMut;
use commonware_codec::{Buf,
    Decode, Encode, EncodeSize, Error, FixedSize, Read, ReadExt,
    ReadRangeExt, Write, RangeCfg
};
use core::ops::RangeInclusive; // Example RangeCfg

// Define a simple configuration for reading Item
// Here, it just specifies the maximum allowed metadata length.
#[derive(Clone)]
pub struct ItemConfig {
    max_metadata_len: usize,
}

// Define a custom struct
#[derive(Debug, Clone, PartialEq)]
struct Item {
    id: u64,           // FixedSize
    name: Option<u32>, // EncodeSize (depends on Option)
    metadata: Vec<u8>, // EncodeSize (variable)
}

// 1. Implement Write
impl Write for Item {
    fn write(&self, buf: &mut impl BufMut) {
        self.id.write(buf);       // u64 implements Write
        self.name.write(buf);     // Option<u32> implements Write
        self.metadata.write(buf); // Vec<u8> implements Write
    }
}

// 2. Implement EncodeSize
impl EncodeSize for Item {
    fn encode_size(&self) -> usize {
        // Sum the sizes of the parts
        self.id.encode_size()         // u64 implements EncodeSize (via FixedSize)
        + self.name.encode_size()     // Option<u32> implements EncodeSize
        + self.metadata.encode_size() // Vec<u8> implements EncodeSize
    }
}

// 3. Implement Read
impl Read for Item {
    type Cfg = ItemConfig;
    fn read_cfg(buf: &mut impl Buf, cfg: &ItemConfig) -> Result<Self, Error> {
        // u64 requires Cfg = (), uses ReadExt::read
        let id = <u64>::read(buf)?;

        // Option<u32> requires Cfg = (), uses ReadExt::read
        let name = <Option<u32>>::read(buf)?;

        // For Vec<u8>, the required config is (RangeCfg, InnerConfig)
        // InnerConfig for u8 is (), so we need (RangeCfg, ())
        // We use ReadRangeExt::read_range which handles the () for us.
        // The RangeCfg limits the vector length using our ItemConfig.
        let metadata_range = 0..=cfg.max_metadata_len; // Create the RangeCfg
        let metadata = <Vec<u8>>::read_range(buf, metadata_range)?;

        Ok(Self { id, name, metadata })
    }
}

// Now you can use Encode and Decode:
let item = Item { id: 101, name: None, metadata: vec![1, 2, 3] };
let config = ItemConfig { max_metadata_len: 1024 };

// Encode the item (uses Write + EncodeSize)
let bytes = item.encode(); // Returns BytesMut

// Decode the item
// decode_cfg ensures all bytes are consumed.
let decoded_item = Item::decode_cfg(bytes, &config).unwrap();
assert_eq!(item, decoded_item);

Re-exports§

pub use config::RangeCfg;
pub use error::Error;
pub use mode::InvalidMode;
pub use mode::Mode;
pub use mode::Modes;
pub use codec::*;
pub use extensions::*;

Modules§

codec
Core traits for encoding and decoding.
config
Types for use as crate::Read::Cfg.
error
Error types for codec operations.
extensions
Extension traits for ergonomic operations on encoding and decoding.
mode
Compact encoding for ordered, extensible mode values.
types
Codec implementations for common types
util
Helpers for reading, validating, and sizing encoded values.
varint
Variable-length integer encoding and decoding

Macros§

mode
Creates a Mode from a u8 literal or expression.
modes
Creates a canonical Modes packet from values convertible to Mode.

Structs§

Copying
An adapter for decoding from a borrowed byte slice.
Viewing
An adapter for decoding from borrowed Bytes.

Traits§

Buf
A buffer accepted by codec readers.
Input
An input that can be consumed by a decoder.
OwnedBytes
An owned byte container a decoded field can be built in.

Derive Macros§

FixedArray
Derives byte-array conversion impls for a fixed-size type.