qubit-codec 0.17.4

Core codec traits and buffer conversion primitives for Rust
Documentation

Qubit Codec (rs-codec)

Rust CI Coverage Crates.io Rust License 中文文档

qubit-codec addresses a recurring problem in format and adapter crates: once a team defines how one logical value maps to encoded units, it still needs owned one-shot APIs, caller-buffered streaming, and optional policy hooks. Rebuilding indices, capacity bounds, progress, and reset/finish lifecycle in each layer produces incompatible public surfaces for the same representation rules. This crate centralizes those mechanics while the format crate keeps domain encoding and errors. It does not ship concrete binary formats, character sets, Base64, hex, percent encoding, or general-purpose std::io extensions.

A binary format crate example

A protocol crate must encode 0x1234 as [0x12, 0x34], decode the reverse, treat a lone input byte as incomplete instead of entering the unsafe decode entry, and encode a batch of values into a caller-owned buffer—all from one Codec implementation. Callers that only need a single owned conversion use CodecValueEncoder / CodecValueDecoder; callers that own the destination buffer use CodecTranscodeEncoder without duplicating bound arithmetic.

Installation

[dependencies]
qubit-codec = "0.17"

The default feature set is empty. Enable json for generic Serde JSON value codecs, io for the qubit-io buffered bridges, and registry for global value codec registration and the register_value_string_codec! / register_value_bytes_codec! macros. The features are independent:

[dependencies]
qubit-io = "0.19"
qubit-codec = { version = "0.17", features = ["io"] }

[dependencies]
qubit-codec = { version = "0.17", features = ["registry"] }

[dependencies]
qubit-codec = { version = "0.17", features = ["json"] }

With json, the generic codecs encode and decode complete JSON values as strings or bytes. For example:

use qubit_codec::{JsonBytesValueCodec, JsonStringValueCodec, ValueDecoder, ValueEncoder};

let mut text = JsonStringValueCodec::<String>::new();
let encoded = text.encode(&String::from("hello")).unwrap();
assert_eq!(text.decode(&encoded).unwrap(), "hello");
let mut bytes = JsonBytesValueCodec::<String>::new();
let encoded = bytes.encode(&String::from("hello")).unwrap();
assert_eq!(bytes.decode(&encoded).unwrap(), "hello");

T must implement serde::Serialize for encoding and serde::de::DeserializeOwned for decoding; both directions return serde_json::Error. These are generic Serde JSON codecs, not concrete domain formats. With both json and registry, the built-in IDs register only serde_json::Value; a custom DTO needs its own descriptor and unique registry ID.

Quick start

Implement the representation rule once, then expose owned encoding:

use core::{convert::Infallible, num::NonZeroUsize};
use qubit_codec::{Codec, CodecValueEncoder, DecodeFailure, ValueEncoder};

#[derive(Default)]
struct U16BeCodec;

impl Codec for U16BeCodec {
    type Value = u16;
    type Unit = u8;
    type DecodeError = Infallible;
    type EncodeError = Infallible;

    const MIN_UNITS_PER_VALUE: usize = 2;
    const MAX_ENCODE_UNITS_PER_VALUE: usize = 2;
    const MAX_DECODE_UNITS_PER_VALUE: usize = 2;

    unsafe fn decode(
        &mut self,
        input: &[u8],
        input_index: usize,
    ) -> Result<(u16, NonZeroUsize), DecodeFailure<Infallible>> {
        debug_assert!(input_index + 2 <= input.len());
        let value = u16::from_be_bytes([
            input[input_index],
            input[input_index + 1],
        ]);
        Ok((value, NonZeroUsize::new(2).expect("two is non-zero")))
    }

    unsafe fn encode(
        &mut self,
        value: &u16,
        output: &mut [u8],
        output_index: usize,
    ) -> Result<usize, Infallible> {
        debug_assert!(output_index + 2 <= output.len());
        output[output_index..output_index + 2]
            .copy_from_slice(&value.to_be_bytes());
        Ok(2)
    }
}

let mut encoder = CodecValueEncoder::new(U16BeCodec);
let bytes = encoder.encode(&0x1234).expect("encoding is infallible");
assert_eq!(vec![0x12, 0x34], bytes);

The lifecycle-aware decode example shows when CodecValueDecoder::decode_lifecycle is required instead of strict one-value decode. The user guide continues this fixed-width scenario with owned decode, incomplete input, caller-buffered streaming, policy hooks, and optional qubit-io integration.

The same U16BeCodec can back both owned helpers and batch encoding into a caller buffer:

use qubit_codec::{
    CodecValueDecoder, CodecValueEncoder, CodecTranscodeEncoder, Transcoder, ValueDecoder,
    ValueEncoder,
};

fn encode_u16(value: u16) -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    let mut encoder = CodecValueEncoder::new(U16BeCodec);
    Ok(encoder.encode(&value)?)
}

fn decode_u16(bytes: &[u8]) -> Result<u16, Box<dyn std::error::Error>> {
    let mut decoder = CodecValueDecoder::new(U16BeCodec);
    Ok(decoder.decode(bytes)?)
}

fn encode_batch(values: &[u16]) -> Result<Vec<u8>, Box<dyn std::error::Error>> {
    let mut encoder = CodecTranscodeEncoder::new(U16BeCodec);
    let capacity = encoder
        .max_total_output_len(values.len())
        .expect("capacity arithmetic should not overflow");
    let mut output = vec![0_u8; capacity];
    let written = encoder.transcode_complete_into(values, &mut output)?;
    output.truncate(written);
    Ok(output)
}

assert_eq!(encode_u16(0x1234)?, vec![0x12, 0x34]);
assert_eq!(decode_u16(&[0x12, 0x34])?, 0x1234);
assert_eq!(
    encode_batch(&[0x1234, 0xabcd])?,
    vec![0x12, 0x34, 0xab, 0xcd],
);

TranscodeEncodeOutput::transcode keeps this whole-range behavior and reserves the batch's full output bound before encoding. For large inputs, the io feature also provides transcode_step, which calls the encoder once and returns TranscodeProgress so the caller can advance through the input while handling output backpressure. The stepwise path limits each call's initial reservation to a single-value bound, but retained buffer capacity does not shrink and a custom encoder may request more space. Existing buffered writers do not adopt this path automatically.

Checked adapters establish index and capacity preconditions before the unsafe Codec entry points; a one-byte input therefore surfaces as incomplete or transcode failure rather than undefined behavior. See the user guide for incremental transcode, EOF handling, and engines with hooks.

What it provides

  • Core value/unit contract: Codec, DecodeFailure, and explicit encode/decode bounds for checked adapters.
  • Owned whole-value facades: ValueEncoder, ValueDecoder, CodecValueEncoder, CodecValueDecoder, including lifecycle-aware decode when reset or finish values are declared.
  • Strict caller-buffered conversion: Transcoder, CodecTranscodeEncoder, CodecTranscodeDecoder, CodecTranscodeConverter, with TranscodeProgress and TranscodeStatus for backpressure.
  • Policy-aware engines: engine::TranscodeEncodeEngine, engine::TranscodeDecodeEngine, engine::TranscodeConvertEngine, and hooks such as DecodeIncompleteAction for incomplete tails after EOF is known.
  • Byte order helpers: ByteOrder, ByteOrderSpec, BigEndian, LittleEndian, NativeEndian.
  • With registry: validated stable IDs, ValueStringCodecDescriptor, ValueBytesCodecDescriptor, ValueStringCodecRegistry, ValueBytesCodecRegistry, and register_value_string_codec! / register_value_bytes_codec! for type-checked erased whole-value codecs (string or bytes wire) fixed at registration time. Registered codec errors must implement Error + Send + Sync + 'static; source errors are retained across threads, while decoded values remain Box<dyn Any> and the whole decode result is not thereby transferable across threads.
  • With io: sync and partial-I/O async transcode input/output bridges over qubit-io.

The crate does not implement domain formats or character-set tables on its own. Codec::decode separates incomplete visible prefixes from invalid domain input through DecodeFailure; an incomplete prefix is not an EOF decision by itself. In the same codec state, Codec::encode_len must match a successful Codec::encode, including intentional zero-output buffering. A Transcoder follows reset -> transcode/transcode_eof -> finish; NeedInput leaves its tail with the caller, NeedOutput requires more destination capacity, and Complete means all visible input from the call's start index was consumed. Capacity bounds must cover every reachable transient state; owned adapters may allocate while streaming APIs use caller-provided buffers. Registry lookup is by validated ID and execution checks the erased input type before invoking user code.

When a synchronous input bridge is driven through EOF with output capacity remaining, it dispatches transcode_eof even if no unread input remains. Async callers dispatch the same EOF event with transcode_eof_step after EndOfInput.

Learn more

Testing

# Run tests with the default feature set
cargo test

# Run tests with all declared features
cargo test --all-features

# Project CI checks
./.infra/bin/ci-check.sh

# Check code coverage
./.infra/bin/coverage.sh

License

Copyright (c) 2025 - 2026. Haixing Hu. All rights reserved.

Licensed under the Apache License, Version 2.0. See LICENSE for the full license text.

Contributing

Contributions are welcome. Please follow the Rust API guidelines, keep public API documentation and tests current, and run ./.infra/bin/align-ci.sh to format code and ./.infra/bin/ci-check.sh to satisfy CI requirements before submitting a pull request.

Author

Haixing Hu - Qubit Co. Ltd.

Repository: https://github.com/qubit-ltd/rs-codec