Crate lencode 

Compact, fast binary encoding with varints and optional deduplication.

This crate provides two core traits, Encode and Decode, for serializing types to a Write and deserializing from a Read without relying on std. Integer types use a compact variable‑length scheme (see Lencode) that encodes small values in a single byte while remaining efficient for large values.

Optional deduplication can be enabled per encode/decode call via DedupeEncoder/DedupeDecoder, which replaces repeated values with small IDs to reduce size for data with many duplicates.

Derive macros for Encode, Decode, and Pack are available from the companion crate lencode_macros and re‑exported in prelude. #[derive(Pack)] on a #[repr(transparent)] single‑field struct automatically generates bulk pack_slice and unpack_vec overrides for zero‑copy I/O.

Bytes and strings are compacted using a flagged header with opportunistic zstd compression:

• Formats: &[u8], Vec<u8>, VecDeque<u8>, &str, String
• Wire: varint((payload_len << 1) | flag) + payload
  • flag = 1 → payload is a zstd frame (original size is stored in the frame)
  • flag = 0 → payload is raw bytes/UTF‑8
• The encoder picks whichever is smaller per value.
• High‑entropy data (random bytes) is detected via a fast entropy check and skips compression entirely, avoiding wasted work.

This keeps headers minimal while improving size significantly for repetitive content, and is no_std compatible via zstd-safe.

Incremental diff encoding

DiffEncoder/DiffDecoder provide stateful delta encoding for keyed byte blobs. When a blob is re-encoded under the same key, only the diff is emitted. Two strategies are tried and the smaller output is picked automatically:

• RLE patches (mode 1): run-length-encoded list of changed regions
• XOR + zstd (mode 2): XOR old and new blobs, then zstd-compress the result

Supported byte types: Vec<u8>, &[u8], [u8; N], VecDeque<u8>. Enable via EncoderContext::with_diff / DecoderContext::with_diff and call set_key() before each encode/decode to opt in for a given field.

Both DedupeEncoder and DiffEncoder/DiffDecoder provide introspection methods (len(), num_keys(), memory_usage(), etc.) and granular state management (clear(), clear_type::<T>(), remove_key()).

Bulk encoding

Collections of fixed‑size elements (e.g. Vec<[u8; 32]>) are encoded via bulk memcpy when possible, avoiding per‑element overhead. This is handled automatically through Encode::encode_slice and Decode::decode_vec, which types can override for optimized batch operations. The Pack trait provides analogous Pack::pack_slice/Pack::unpack_vec methods for deduplicated types.

Quick start:

use lencode::prelude::*;

#[derive(Encode, Decode, PartialEq, Debug)]
struct Point { x: u64, y: u64 }

let p = Point { x: 3, y: 5 };
let mut buf = Vec::new();
let _n = encode(&p, &mut buf).unwrap();
let q: Point = decode(&mut Cursor::new(&buf)).unwrap();
assert_eq!(p, q);

Collections and primitives:

use lencode::prelude::*;

let values: Vec<u128> = (0..10).collect();
let mut buf = Vec::new();
encode(&values, &mut buf).unwrap();
let roundtrip: Vec<u128> = decode(&mut Cursor::new(&buf)).unwrap();
assert_eq!(values, roundtrip);

Deduplication (smaller output for repeated values):

use lencode::prelude::*;

// A small type we want to dedupe; implements Pack and the dedupe markers.
// Note that this is a toy example, in practice `MyId` would be more
// efficiently encoded using regular lencode encoding because it wraps a u32.
#[derive(Clone, Copy, PartialEq, Eq, Debug, Hash)]
struct MyId(u32);

impl Pack for MyId {
    fn pack(&self, w: &mut impl Write) -> Result<usize> { self.0.pack(w) }
    fn unpack(r: &mut impl Read) -> Result<Self> { Ok(Self(u32::unpack(r)?)) }
}
impl DedupeEncodeable for MyId {}
impl DedupeDecodeable for MyId {}

// Prepare some data with many repeats
let vals = vec![MyId(42), MyId(7), MyId(42), MyId(7), MyId(42), MyId(7), MyId(42)];

// Encode without deduplication
let mut plain = Vec::new();
encode(&vals, &mut plain).unwrap();

// Encode with deduplication enabled
let mut ctx = EncoderContext::with_dedupe();
let mut deduped = Vec::new();
encode_ext(&vals, &mut deduped, Some(&mut ctx)).unwrap();
assert!(deduped.len() < plain.len());

// Round-trip decoding with a DecoderContext
let mut ctx_dec = DecoderContext::with_dedupe();
let rt: Vec<MyId> = decode_ext(&mut Cursor::new(&deduped), Some(&mut ctx_dec)).unwrap();
assert_eq!(rt, vals);

Modules

context

Unified encoding/decoding context that bundles optional deduplication and diff state.

dedupe

diff

Incremental binary diff encoding/decoding for byte blobs.

io

Lightweight, no-std compatible I/O traits and adapters used by the Encode/Decode APIs.

pack

prelude

Convenience re‑exports for common traits, modules and derive macros.

tuples

u256

A compact U256 newtype with varint and endianness support.

varint

Varint traits and helpers used by the Lencode scheme.

Macros

impl_pack_for_endianness_types

Macro to implement the Pack trait for types that implement endian_cast::Endianness. This avoids orphan rule issues by allowing explicit implementations per type.

impl_to_unsigned_signed

Implements ToUnsigned/ToSigned pairwise conversions for signed/unsigned primitives.

impl_unsigned_integer

Implements the internal integer helper traits for the given unsigned types.

Traits

Decode

Trait for types that can be decoded from a binary stream.

Encode

Trait for types that can be encoded to a binary stream.

Functions

decode

Decodes a value of type T from reader using T’s Decode implementation.

decode_ext

Decodes a value with an optional DecoderContext for deduplication and/or diff decoding.

encode

Encodes value into writer using the type’s Encode implementation.

encode_ext

Encodes value with an optional EncoderContext for deduplication and/or diff encoding.

Type Aliases

Result

Crate‑wide Result that defaults to Error.