Qubit IO Binary
qubit-io-binary lets Rust applications read and write a structured binary
protocol without coupling the protocol to a file type or async runtime. It
adapts the buffer codecs in qubit-codec-binary to qubit-io streams:
- synchronous extensions use
ByteInputandByteOutput; - asynchronous extensions use
AsyncInput<Item = u8>andAsyncOutput<Item = u8>; - both layers share the same fixed-width, LEB128, ZigZag, and UTF-8 codec semantics;
- typed synchronous readers and writers can carry byte order or decode policy;
- buffered synchronous wrappers amortize repeated small operations.
The core API does not depend on Tokio, futures-io, or another executor.
qubit-io supplies opt-in adapters for ecosystem stream traits.
Installation
[]
= "0.8"
The quick start below needs only qubit-io-binary. Add qubit-codec or
qubit-codec-binary when importing their byte-order or decode-policy types,
and add qubit-io when writing generic ByteInput or ByteOutput bounds.
Quick Start: a Synchronous Record
Standard Read and Write implementations, including Cursor and Vec<u8>,
implement the qubit-io abstractions through adapters.
use Cursor;
use BinaryReadExt;
use BinaryWriteExt;
use Leb128ReadExt;
use Leb128WriteExt;
Runtime-Neutral Async
Async method names use the _async suffix so a type may expose both sync and
async stream APIs without ambiguity.
use ;
use ;
async
This is a compile-only async function example. Call it from an executor in your application; the README fixture does not run it.
Asynchronous extension methods only require Unpin; they do not impose a
runtime or Send requirement. These operations are not cancellation safe:
dropping a pending read retains bytes already consumed, and dropping a pending
write leaves any already-written prefix in the output.
What It Provides
| Encoding | Synchronous traits | Asynchronous traits |
|---|---|---|
| Fixed-width scalars | BinaryReadExt, BinaryWriteExt |
AsyncBinaryReadExt, AsyncBinaryWriteExt |
| LEB128 | Leb128ReadExt, Leb128WriteExt |
AsyncLeb128ReadExt, AsyncLeb128WriteExt |
| ZigZag | ZigZagReadExt, ZigZagWriteExt |
AsyncZigZagReadExt, AsyncZigZagWriteExt |
| Length-prefixed UTF-8 | StringReadExt, StringWriteExt |
AsyncStringReadExt, AsyncStringWriteExt |
Synchronous typed wrappers include BinaryReader, BinaryWriter,
Leb128Reader, Leb128Writer, ZigZagReader, and ZigZagWriter, together
with buffered variants. They are generic over Input and Output; they are not
defined in terms of std::io::Read and std::io::Write.
Varint readers require an explicit Strict or NonStrict policy type. Strict
readers keep the short method names such as read_u64; permissive readers use
the explicit read_u64_non_strict (and corresponding string) methods.
Buffered readers provide into_parts to recover the wrapped input together
with unread prefetched bytes. Buffered writers provide into_parts to recover
the wrapped output and a BufferedOutputRecovery. Pending bytes were not
reported as written; Indeterminate means a panicking inner write may have
produced an external prefix, so do not replay it automatically. These methods
perform no I/O: flush first for normal completion, and retain the writer to
retry a flush failure.
Strict LEB128 methods reject non-canonical encodings. String read methods accept
a maximum payload length to bound allocation. For persistent formats, prefer
fixed-width length fields or u64 LEB128 lengths over target-width usize
helpers.
High-throughput string readers can call read_utf8_payload_into (or the async
variant) with a reusable Vec<u8> to avoid reallocating payload storage for
each record. The buffer is retained on invalid UTF-8 errors for diagnostics.
Mixed Binary Stream Benchmark
The 2026-08-04 Criterion run below used a deterministic random mix of u8,
i32, u64, and multibyte UTF-8 strings (131,072 fields per iteration). The
full command and environment are recorded in
benches/results/2026-08-04-mixed-binary-pipeline.md.
The reusable payload-buffer comparison is recorded in
benches/results/2026-08-07-string-buffer-reuse.md.
The in-memory mixed-pipeline comparison is recorded in
benches/results/2026-10-24-memory-mixed-binary-pipeline-correction.md.
Treat these values as a comparison of buffering strategies, not as portable
performance guarantees.
Buffering is most useful when small operations would otherwise reach a costly
underlying stream. For in-memory streams such as Cursor, or streams that are
already buffered, start by comparing the extension path before adding another
buffer. The linked in-memory correction report
shows that the buffered path was slower in that run; it is not a universal
ranking. See the user guide for selection guidance.
The benchmark suite is split into layers: micro_binary_pipeline uses
short-write in-memory streams, memory_mixed_binary_pipeline compares mixed
records without filesystem effects, and the prod_* groups measure file-backed
end-to-end pipelines. async_binary_pipeline separately measures the
runtime-neutral asynchronous adapters. Set QUBIT_IO_STREAM_BENCH_GROUP to run
one group at a time; this keeps adapter cost, buffering cost, and filesystem
cost distinguishable.
| Scenario | Time | Throughput | Relative to raw extension |
|---|---|---|---|
| Write: raw extension | 65.19 ms | 2.01 M fields/s | 1.0× |
Write: extension + BufWriter |
4.04 ms | 32.44 M fields/s | 16.1× |
Write: BufferedBinaryWriter |
3.70 ms | 35.40 M fields/s | 17.6× |
| Read: raw extension | 46.80 ms | 2.80 M fields/s | 1.0× |
Read: extension + BufReader |
3.64 ms | 36.01 M fields/s | 12.9× |
Read: BufferedBinaryReader |
3.46 ms | 37.90 M fields/s | 13.5× |
The buffered wrappers were about 9% faster for writes and 5% faster for reads than the corresponding externally buffered extension paths in this run.
Boundaries and Further Reading
qubit-codec-binaryowns buffer-level binary algorithms.qubit-ioowns generic synchronous and runtime-neutral asynchronous streams.qubit-io-binarycomposes the two without owning files or runtimes.
Use this crate for binary values on an existing stream. It does not open files or own an async runtime. For a scenario-led tutorial, see the user guide or 中文用户指南; for every public item, see the API reference.
Testing
# Run tests with the default feature set
# Run the README quick-start fixture with its checked-in lockfile
# Run tests with all declared features
# Project CI checks
# Check code coverage
License
Copyright (c) 2025 - 2026. Haixing Hu. All rights reserved.
Licensed under the Apache License, Version 2.0. See LICENSE for the full license text.
Contributing
Contributions are welcome. Please follow the Rust API guidelines, keep public
API documentation and tests current, and run ./.infra/bin/align-ci.sh to format code and
./.infra/bin/ci-check.sh to satisfy CI requirements before submitting a pull request.
Author
Haixing Hu - Qubit Co. Ltd.
Repository: https://github.com/qubit-ltd/rs-io-binary