Qubit Codec

Core codec traits and buffer conversion primitives for Rust.

Overview

Qubit Codec is the domain-neutral foundation for Qubit codec crates. It contains small traits and value types that are shared by binary, text, misc, and I/O adapter crates, without concrete format implementations.

This crate provides:

• Codec for low-level single-value buffer codecs.
• CodecValueExt, CodecValueEncoder, CodecValueDecoder, CodecTranscodeEncoder, CodecTranscodeDecoder, and CodecTranscodeConverter adapters for explicit codec-backed value and buffered conversion.
• TranscodeEncodeEngine, TranscodeEncodeHooks, EncodeOutcome, and EncodeContext for reusing the common buffered encoding loop in policy-aware downstream encoders.
• TranscodeDecodeEngine, TranscodeDecodeHooks, DecodeInvalidAction, and DecodeContext for reusing the common buffered decoding loop in policy-aware downstream decoders.
• TranscodeConvertEngine and TranscodeConvertEngineError for policy-aware unit-to-unit conversion pipelines built from a decode side and an encode side.
• ValueEncoder and ValueDecoder traits for owned whole-value convenience APIs.
• Transcoder, TranscodeProgress, and TranscodeStatus for caller-managed logical-stream conversion.
• TranscodeEncoder, TranscodeDecoder, and TranscodeConverter marker traits for semantic transcoder direction.
• ByteOrder, ByteOrderSpec, BigEndian, and LittleEndian for byte-order metadata shared by binary and text codecs.

Concrete codecs live in sibling crates such as qubit-codec-binary, qubit-codec-text, and qubit-codec-misc.

Design Goals

• Layered Boundaries: keep domain-neutral traits separate from binary, text, misc, and stream-specific implementations.
• Small Public Surface: expose only the primitives that multiple codec crates need to share.
• Policy Neutrality: leave charset, malformed-input, and wire-format rules to domain crates.
• Zero-Cost Markers: represent byte order as copyable type/value markers without runtime allocation.
• Stable Progress Reporting: use TranscodeProgress and TranscodeStatus to make caller-managed buffer conversion explicit.

Features

Core Conversion Traits

• Codec: encodes and decodes one value or codec quantum against a caller-managed unit buffer.
• DecodeFailure: separates incomplete-prefix flow control from codec-domain invalid input returned by Codec::decode.
• CodecEncodeError / CodecDecodeError / CodecConvertError: add adapter-level encode, decode, and conversion errors without hiding codec-specific failures. Buffer index and capacity failures are represented by TranscodeError.
• ValueEncoder<Input>: converts a borrowed value into an owned output type.
• ValueDecoder<Input>: converts a borrowed encoded value into an owned decoded output type.
• CodecValueEncoder<C>: wraps a Codec as a ValueEncoder<C::Value> that returns owned Vec<C::Unit> output.
• CodecValueDecoder<C>: wraps a Codec as a ValueDecoder<[C::Unit]> that accepts exactly one encoded value.
• CodecValueExt: extension trait for checked one-value codec helpers such as reset-prefixed encode and exact decode with flush handling.

Buffered Transcoder Primitives

• Transcoder<Input, Output>: converts input units into output units inside caller-provided buffers, then finishes internally retained output after the caller has handled any incomplete input tail.
• TranscodeEncoder<Value, Unit>: semantic Transcoder bound for value-to-unit buffered encoding.
• TranscodeDecoder<Unit, Value>: semantic Transcoder bound for unit-to-value buffered decoding.
• TranscodeConverter<InputUnit, OutputUnit>: semantic Transcoder bound for unit-to-unit buffered conversion.
• CodecTranscodeEncoder<C>: wraps a Codec as a TranscodeEncoder<C::Value, C::Unit> over caller-provided output buffers.
• TranscodeEncodeEngine<C, H>: reusable engine that owns a codec plus policy hooks and runs the common buffered encoding loop.
• TranscodeEncodeHooks<C>: policy hook trait used by codec-backed encoders that need custom transcode/finalization behavior while sharing the common loop.
• EncodeOutcome: outcome returned by encode hooks for one value: consumed with a written count, or not consumed because more output is needed.
• EncodeContext<'a, Value, Unit>: input value, input index, output slice, and cursor passed to encode hooks.
• CodecTranscodeDecoder<C>: wraps a Codec as a strict TranscodeDecoder<C::Unit, C::Value> that leaves engine-detected incomplete tails in the caller's input buffer and wraps codec-reported decode errors.
• TranscodeDecodeEngine<C, H>: reusable engine that owns a codec, policy hooks, and the common decode loop.
• TranscodeDecodeHooks<C>: policy hook trait used by codec-backed decoders that need custom invalid-input behavior while sharing the common decode loop.
• DecodeInvalidAction<Value>: hook return value used by decoder engines for invalid-input policy decisions.
• CodecTranscodeConverter<D, E>: composes a decoding codec and an encoding codec as a policy-free TranscodeConverter.
• TranscodeConvertEngine<D, E, DH, EH>: reusable unit-to-unit converter engine that composes decode hooks, encode hooks, and the common buffered conversion loop.
• TranscodeDecodeInput<I>: owns a unit-level BufferedInput and drives caller-provided streaming decoders through transcode_into / finish_transcode_into.
• TranscodeEncodeOutput<O>: owns a unit-level BufferedOutput; ordinary flush drains buffered units. Stateful streaming encoders use transcode_from and finish.
• TranscodeProgress: reports relative input units read and output units written.
• TranscodeStatus: distinguishes complete conversion from NeedInput and NeedOutput stops.
• TranscodeError / CapacityError / TranscodeContractError: report framework-level buffer, capacity-planning, and broken-progress contract failures separately from codec or policy domain errors.

Byte Order Markers

• ByteOrder: runtime byte-order enum for public APIs.
• ByteOrderSpec: type-level byte-order trait used by hot codecs.
• BigEndian / LittleEndian: zero-sized marker types.

Focused Public API

• No concrete formats: binary, text, and miscellaneous codecs are published in sibling crates.

Choosing the Right Abstraction

qubit-codec ships several layers because real codec stacks have different needs. Use this decision tree to pick the smallest piece that fits your case.

What are you writing?

├── A new codec for one logical value (a UTF-8 char, a LEB128 integer,
│   a Base64 quantum, a fixed-width scalar, …)
│       → implement Codec
│         (unchecked single-value contract; the foundation everything else builds on)
│
├── A whole-string codec where "one logical value" has no useful meaning
│   (Base64 padding, hex with separators, percent encoding, C string literal, …)
│       → implement ValueEncoder<Input> / ValueDecoder<Input> directly
│         (skip Codec; these two traits also serve as the convenience layer)
│
├── A streaming wrapper around an existing Codec, with no error policy:
│   strict pass-through that surfaces every codec error as-is
│       → use CodecTranscodeDecoder<C> / CodecTranscodeEncoder<C>
│         / CodecTranscodeConverter<D, E>
│         (no custom code; you get a fully wired Transcoder)
│
├── An owned-output wrapper around a Codec (one call → one Vec<Unit>
│   or one Value)
│       → use CodecValueEncoder<C> / CodecValueDecoder<C>
│         (allocates per call; convenience-layer ValueEncoder/Decoder)
│
└── A streaming codec that needs to make decisions on malformed input:
    skip, replace, count, or report — not just propagate
        → implement TranscodeDecodeHooks<C> / TranscodeEncodeHooks<C>
          and wrap them in TranscodeDecodeEngine<C, H> / TranscodeEncodeEngine<C, H>
          (you only write the policy; the engine owns the buffered loop,
           cursor bookkeeping, NeedInput/NeedOutput reporting, and capacity checks)

For unit-to-unit conversion (e.g. UTF-8 bytes → UTF-16 bytes), compose a
decode codec + an encode codec:
- strict pipeline    → CodecTranscodeConverter<D, E>
- with policy hooks  → TranscodeConvertEngine<D, E, DH, EH>

Layer overview

┌────────────────────────────────────────────────────────────────┐
│  qubit-io-binary / qubit-io-text             (concrete I/O)    │
├────────────────────────────────────────────────────────────────┤
│  TranscodeDecodeInput / TranscodeEncodeOutput  (I/O bridges)   │
├────────────────────────────────────────────────────────────────┤
│  TranscodeXxxEngine + TranscodeXxxHooks       (policy + loop)  │
│  CodecTranscodeDecoder / Encoder / Converter  (strict bridges) │
├────────────────────────────────────────────────────────────────┤
│  Transcoder<Input, Output> + TranscodeProgress + TranscodeStatus│
│  ValueEncoder<Input> / ValueDecoder<Input>      (convenience)  │
├────────────────────────────────────────────────────────────────┤
│  Codec                                  (single-value, unchecked) │
└────────────────────────────────────────────────────────────────┘

Implementing further up the stack does not mean rewriting the lower layers: CodecValueEncoder<C> and CodecTranscodeDecoder<C> are concrete adapters that turn any Codec into the higher-layer trait for free. Only drop down to the engine + hooks layer when you actually need policy decisions on invalid input, replacement output, or stateful finish output.

Installation

Add this to your Cargo.toml:

[dependencies]
qubit-codec = "0.9"

Quick Start

use qubit_codec::{
    TranscodeProgress,
    TranscodeStatus,
    ValueEncoder,
};

struct StringEncoder;

impl ValueEncoder<str> for StringEncoder {
    type Output = String;
    type Error = core::convert::Infallible;

    fn encode(&mut self, input: &str) -> Result<Self::Output, Self::Error> {
        Ok(input.to_owned())
    }
}

let mut encoder = StringEncoder;
let encoded = ValueEncoder::<str>::encode(&mut encoder, "codec")?;
assert_eq!("codec", encoded);

let progress = TranscodeProgress::complete(3, 4);
assert_eq!(TranscodeStatus::Complete, progress.status());

# Ok::<(), core::convert::Infallible>(())

API Reference

Core Codec Traits

Codec Adapters

I/O Adapters

Encoder Hooks And Engines

Decoder Hooks And Engines

Converter Engines

Transcoder Operations

TranscodeStatus Values

Contract Notes

• Codec::MIN_UNITS_PER_VALUE is the safety lower bound for calling Codec::decode; Codec::MAX_UNITS_PER_VALUE is the per-value output/read upper bound. Checked adapters assert min <= max before using these values.
• Codec::decode returns DecodeFailure::Incomplete when the visible input is a valid prefix that needs more units, and DecodeFailure::Invalid for codec-domain malformed, non-canonical, or otherwise invalid input.
• encode_len(value) must equal the number of units Codec::encode writes for the same value and codec state, and it must not exceed Codec::MAX_UNITS_PER_VALUE.
• Stateful one-value callers should use CodecValueExt::max_encode_value_units() with CodecValueExt::encode_value_with_reset(), or CodecValueExt::decode_exact_value_with_flush() when the input must contain exactly one encoded value. These helpers keep reset/flush capacity checks and overflow handling in the value adapter layer.
• CodecDecodeError / CodecEncodeError are adapter-level wrappers. TranscodeError is the streaming framework wrapper. Concrete codec, charset, or policy failures remain the associated domain error.
• NeedInput means the reported tail was not consumed and must remain available when the caller retries with more input. It is a streaming boundary signal, not an EOF error; finish does not receive that source tail. Callers must apply their own EOF policy before finalization.
• Default codec-backed decoders and converters are intended for formats whose value boundary is locally decidable from the visible prefix plus codec state. Formats that require EOF-aware maximal-munch parsing, delayed boundary decisions, or reinterpretation of a pending prefix at EOF should use a custom Transcoder or value-level facade for that policy.
• NeedOutput means the reported input was not fully consumed because the output slice reached its bound.

Byte Order Types

Crate Boundary

qubit-codec does not contain concrete binary formats, character sets, or percent/Base64/hex codecs. Its I/O-facing surface is limited to low-level qubit_io::Input / qubit_io::Output bridge types used by downstream stream crates. Keep std::io::Read / std::io::Write extension traits and concrete reader/writer adapters in domain crates so downstream users can depend on only the layers they need.

Performance Considerations

Core traits and buffered adapters do not require heap allocation. BigEndian and LittleEndian are zero-sized, and ByteOrder is a small copyable enum. CodecValueEncoder allocates owned Vec<Unit> output because that is the ValueEncoder contract; concrete downstream codecs may have their own allocation behavior.

Testing & Code Coverage

This project keeps the core trait contracts covered by integration tests under tests/.

Running Tests

# Run all tests
cargo test

# Run with coverage report
./coverage.sh

# Generate text format report
./coverage.sh text

# Align code with CI requirements
./align-ci.sh

# Run CI checks (format, clippy, test, coverage, audit)
RS_CI_SKIP_TOOLCHAIN_UPDATE=1 ./ci-check.sh

Dependencies

Runtime dependencies are intentionally small:

• thiserror provides public error type implementations.
• qubit-io provides BufferedInput and BufferedOutput used by TranscodeDecodeInput and TranscodeEncodeOutput.

License

Copyright (c) 2026. Haixing Hu.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

See LICENSE for the full license text.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Development Guidelines

• Keep this crate free of concrete format implementations.
• Document public traits and marker types with examples.
• Keep tests comprehensive and deterministic.
• Ensure all checks pass before submitting a PR.

Author

Haixing Hu

Related Projects

• qubit-codec-binary: binary buffer codecs.
• qubit-codec-text: charset and Unicode buffer codecs.
• qubit-codec-misc: reusable miscellaneous byte and text codecs.
• qubit-io: generic std::io helpers.
• More Rust libraries from Qubit are available under the qubit-ltd GitHub organization.

Repository: https://github.com/qubit-ltd/rs-codec