smol-bytes 0.2.0

High-performance, clone-efficient byte buffers optimized for small data.

Introduction

smol-bytes provides byte buffers that store up to 62 bytes inline — no heap allocation, and cloning an inline value is a plain 64-byte copy. Larger values use bytes-backed heap storage with reference-counted clones. Two strategies control what happens when a heap value shrinks back under the inline threshold, and UTF-8 wrappers layer String-like, boundary-checked APIs over the same storage.

This is a good fit when most values are small and cloned often — tokens in a lexer, keys and field names, protocol headers — and allocation pressure matters.

Install

[dependencies]
smol-bytes = "0.2"

Python (pip install smol-bytes) and JavaScript (npm install smol-bytes) packages are also published; build-from-source instructions for each binding are below.

Quick start

The root Bytes type is an alias for shared::Bytes. Ordinary copies and static values of at most 62 bytes start inline; imported or owner-backed values and retained shared heap views can remain heap-backed below that threshold.

use smol_bytes::{Bytes, Utf8Bytes};

let small = Bytes::from_static(b"identifier");
assert!(small.is_inline());

let cloned = small.clone(); // 64-byte copy, no allocation
assert_eq!(small, cloned);

let heap = Bytes::copy_from_slice(&[0_u8; 63]);
assert!(heap.is_heap());
assert_eq!(heap.len(), 63);

let text: Utf8Bytes = "identifier".parse().expect("infallible parse");
assert_eq!(text.as_str(), "identifier");

Storage strategies

shared::Bytes preserves heap storage once a value lives there, keeping conversions with bytes::Bytes zero-copy. compact::Bytes copies a shrinking view of 62 bytes or fewer back into inline storage to release the allocation.

use smol_bytes::{compact, shared, Buf};

let mut shared = shared::Bytes::from(vec![0_u8; 100]);
let mut compact = compact::Bytes::from(vec![0_u8; 100]);

shared.advance(70);
compact.advance(70);

assert_eq!(shared.len(), 30);
assert_eq!(compact.len(), 30);
assert!(shared.is_heap());   // stays shareable and zero-copy convertible
assert!(compact.is_inline()); // allocation released, contents inlined

Rule of thumb: use shared (the default) for I/O and bytes interop; use compact when memory footprint matters more than conversion speed.

Types

Every handle is 64 bytes. With the bytes dependency enabled, all byte types implement bytes::Buf, and the mutable ones implement bytes::BufMut.

The immutable strategy types have immutable payloads but mutable views: advance, truncate, and clear change only the visible cursor/window. Utf8Bytes offers the same byte-indexed, UTF-8-boundary-checked view operations and can reclaim unique full heap allocations with try_into_mut. Utf8Buffer and Utf8BytesMut expose as_mut_str, AsMut<str>, BorrowMut<str>, DerefMut<Target = str>, and mutable string-range indexing; the standard str rules preserve UTF-8 validity.

BytesMut::split_to, split_off, and split always return BytesMut. Inline results stay inline; splitting an inline value's reserved-but-unwritten capacity promotes it to heap storage so capacity partitioning remains exact. try_split_to and try_split_off return Result<BytesMut, OutOfBounds> and leave the receiver unchanged on bounds failure. unsplit always consumes and appends its argument, using the zero-copy bytes::BytesMut path for contiguous heap halves and copying/promoting only when required.

Rust UTF-8 split and slice indices are byte offsets that must fall on character boundaries; offenders panic, and the try_split_to, try_split_off, and try_slice variants return errors instead. (The Python bindings differ deliberately — see below.)

The UTF-8 wrappers implement FromStr. Heap-capable wrappers parse infallibly; Utf8Buffer returns TryPutError when text exceeds its fixed inline capacity.

0.2 API migration

0.2 intentionally removes several ambiguous or lossy APIs. For complete Rust, Python, and JavaScript examples, see the 0.2.0 changelog and migration guide.

• Replace Utf8Buffer::from(text) with Utf8Buffer::try_from(text)? (or Utf8Buffer::try_from_str(text)?). FromStr remains available. This makes the 62-byte fixed-capacity limit explicit.
• BytesMut split methods now return BytesMut directly. Remove an inner match/unwrap around split, split_to, and split_off; use the corresponding try_split_* method when bounds must be reported instead of panicking. try_split is removed, and unsplit now returns ().
• Use Buffer::resize(new_len, value) and Buffer::try_resize(new_len, value) to choose the byte written during extension. Failed try_resize calls are atomic.
• UTF-8 index operations now use Utf8IndexError, preserving whether an invalid character boundary, index, or full range caused the failure. Converting owned Buffer, BytesMut, or raw strategy bytes into a UTF-8 wrapper returns FromUtf8Error<B> on invalid input; call into_bytes() to recover the original allocation and view.
• Variable-width inherent Buffer::try_get_uint{,_le,_ne} and Buffer::try_get_int{,_le,_ne} methods now return TryGetIntegerError, distinguishing an invalid width from missing bytes. The bytes::Buf trait methods retain bytes' own error type and panic contract.
• Utf8Buf and Utf8BufMut are sealed implementation traits. Generic mutable code can use try_push/try_push_str and each type's CapacityError; Utf8Buffer reports TryPutError, while Utf8BytesMut uses Infallible.

bytes interop

Conversions with the bytes crate are zero-copy wherever the representation allows:

• bytes::Bytes -> shared::Bytes shares the allocation (From impl).
• shared::Bytes -> bytes::Bytes reuses heap backing; inline values copy.
• compact::Bytes::from(bytes::Bytes) inlines payloads of at most 62 bytes and shares larger ones.
• BytesMut::freeze_shared / freeze_compact convert without copying heap contents; BytesMut::freeze is the root-default shared conversion.
• Utf8BytesMut::freeze similarly returns the root-default Utf8Bytes; freeze_shared and freeze_compact remain available for explicit targets.
• Consuming shared::Bytes/compact::Bytes and UTF-8 strategy conversions apply the target strategy: large heap values keep their allocation and small values compact when converted to the compact strategy.
• Bytes::try_into_mut and Utf8Bytes::try_into_mut reclaim unique full heap allocations; a shared or sliced view is returned unchanged on failure.

use smol_bytes::{BytesMut, Utf8Bytes, Utf8BytesMut, compact};

let bytes = BytesMut::from(&b"payload"[..]).freeze();
assert_eq!(bytes, b"payload"[..]);

let mut text = Utf8Bytes::from("header: value");
text.advance(8);
assert_eq!(text.as_str(), "value"); // only this view moved

let compact: compact::Bytes = bytes.into();
assert_eq!(compact, b"payload"[..]);

let frozen = Utf8BytesMut::from("done").freeze();
assert_eq!(frozen.as_str(), "done");

Features and MSRV

With no features enabled the crate is no_std and provides the fixed Buffer and Utf8Buffer types. bytes::Buf/BufMut are available once alloc, std, or the compatibility bytes feature is selected; alloc adds the heap-backed types without std.

Byte wrapper JSON schemas describe serialization as array<u8> and accept an array or string under Schemars' deserialization contract. Buffer additionally sets maxItems for array input and maxLength for string input to INLINE_CAP. maxLength is a scalar-count bound; deserialization still enforces the actual UTF-8 byte capacity for multibyte strings. UTF-8 wrappers remain ordinary string schemas.

The sqlx bindings decode by borrowing from the row, so a value of at most 62 bytes is built inline with no heap allocation at all. That costs one thing: PostgreSQL will not lend BYTEA out in a simple (unprepared) query, so code on that path — raw_sql, or a SQL string handed straight to an Executor — has to decode the byte types as Vec<u8> and convert. query, query_as and the query! macros carry an argument list and are therefore prepared, so they are unaffected, as are the UTF-8 types and every MySQL and SQLite path.

Rust 1.95 is the library MSRV, and the bytes dependency floor is 1.10. Development-only test and benchmark dependencies can require a newer compiler, and so do both optional integrations, to different floors: async-graphql 7.2 declares Rust 1.89, and 7.2 is the floor because the 7.0 releases do not build against the 7.2 derive crate their own dependency range admits; sqlx 0.9 declares Rust 1.94, and 0.9 is the floor because the impls are written against the lifetime-free Database::ArgumentBuffer introduced there. Enabling either raises the MSRV for the whole build.

Verification

The test and CI story is deliberately heavier than the crate's size:

• Unit, integration, doc, and property tests (proptest state-machine comparisons against Vec/String, plus quickcheck and arbitrary generators that preserve type invariants).
• Miri over the full suite under both stacked borrows and tree borrows with strict provenance and symbolic alignment checks.
• Address, leak, memory, and thread sanitizers in CI.
• Deserialization is hardened: borsh reads length-prefixed payloads in bounded chunks instead of trusting the length prefix, and serde sequence hints are capped before preallocating.

Python

Install the published package with pip install smol-bytes (Python 3.11+).

To build from a checkout instead, Python 3.11+, Rust, and maturin are required:

python -m venv .venv
source .venv/bin/activate
python -m pip install maturin pytest
maturin develop --features pyo3 --manifest-path smol-bytes-py/Cargo.toml
python -m pytest tests/python -v

The root smol_bytes module exposes canonical shared Bytes and Utf8Bytes alongside Buffer, BytesMut, Utf8Buffer, and Utf8BytesMut. smol_bytes.shared.Bytes and smol_bytes.shared.Utf8Bytes are aliases of those exact root classes (including their root __module__), while smol_bytes.compact remains a distinct compact strategy.

Raw byte classes use from_utf8() and decode_utf8() for explicit text conversion; str(raw) is always the same total representation as str(bytes(raw)). Their from_bytes() and mutable put_slice() methods accept bytes, bytearray, and readable one-dimensional C-contiguous byte memoryview values, copying the input before mutation.

The UTF-8 classes are string-like from Python: len(), indexing, and iteration use Unicode scalar values. Their explicit APIs are char_len(), byte_len(), split_to_chars(), split_off_chars(), slice_chars(), and for mutable/fixed classes truncate_chars(). remaining_bytes() and advance_bytes() are deliberately byte-oriented, as are get_* methods.

from smol_bytes import Bytes, BytesMut, Utf8Bytes

raw = Bytes.from_bytes(b"abc")
assert raw.is_inline()
assert bytes(raw) == b"abc"
assert raw.decode_utf8() == "abc"

text = Utf8Bytes.from_str("café")
assert len(text) == 4        # Unicode characters
assert text.char_len() == 4
assert text.byte_len() == 5  # UTF-8 bytes
assert str(text) == "café"
assert str(text.split_to_chars(3)) == "caf"

mutable = BytesMut.from_bytes(bytearray(b"abc"))
frozen = mutable.freeze_shared()
assert bytes(mutable) == b""  # source remains reusable
assert bytes(frozen.to_mutable()) == b"abc"

Binding behavior worth knowing:

• Methods that allocate proportionally to caller data raise MemoryError on absurd or failing requests instead of aborting the interpreter, like CPython containers.
• memoryview(...) over the shared and compact Bytes classes exports a snapshot copy, not a live view.
• Slice assignment on the mutable classes requires matching lengths, and contiguous assignments take a direct copy fast path.
• Python unsplit() uses non-consuming snapshot semantics: it appends the call-entry contents while leaving the argument unchanged. Passing the same mutable object as both operands raises RuntimeError atomically.

JavaScript / WebAssembly

Install the published package with npm install smol-bytes.

To build from a checkout instead, install Node.js 20, wasm-pack 0.13.1, and the wasm32-unknown-unknown target. wasm-pack and Cargo resolve a compatible wasm-bindgen 0.2.x release; any separately installed wasm-bindgen helper must match the resolved crate version. CI checks the supported 0.2.x family.

rustup target add wasm32-unknown-unknown
cd js
npm ci
npm run build
npm test

The Wasm build entry point is:

wasm-pack build smol-bytes-wasm --target bundler --out-dir ../js/pkg --out-name smol_bytes -- --features wasm

The root export canonically contains Bytes and Utf8Bytes using the shared strategy, alongside Buffer, BytesMut, Utf8Buffer, and Utf8BytesMut. smol-bytes/shared exports the exact same immutable class objects; the smol-bytes/compact entry point exports distinct compact classes. Every entry point initializes iteration, disposal, and input validation independently.

Raw bytes use fromUtf8() and decodeUtf8(); they deliberately have no string-like fromString() or raw toString() API. Neither raw nor UTF-8 public classes expose fromStatic(); UTF-8 classes retain fromString() / toString() and make byte units explicit: byteLength(), charLength(), remainingBytes(), advanceBytes(), splitToBytes(), splitOffBytes(), sliceBytes(), and truncateBytes(). Byte offsets must land at UTF-8 scalar boundaries. Numeric arguments are checked before WebAssembly coercion, and malformed JavaScript UTF-16 strings are rejected before encoding.

Byte conversions at the Wasm boundary return copies. clone() is available on each public byte/string value class. Mutable freezeShared() / freezeCompact() drain their source into the selected immutable strategy, while immutable toMutable() returns an independent mutable value. JavaScript unsplit() snapshots and appends other without consuming it; self-aliasing deterministically doubles the value.

import { Bytes, BytesMut, Utf8Bytes } from "smol-bytes";
import { Bytes as CompactBytes } from "smol-bytes/compact";

const raw = CompactBytes.fromBytes(new Uint8Array([1, 2, 3]));
console.assert(raw.isInline());
console.assert(raw.toBytes()[2] === 3);

const shared = Bytes.fromUtf8("hello");
console.assert(shared.decodeUtf8() === "hello");

const mutable = BytesMut.fromUtf8("ready");
const frozen = mutable.freezeShared();
console.assert(mutable.isEmpty() && frozen.decodeUtf8() === "ready");

const text = Utf8Bytes.fromString("café");
console.assert(text.byteLength() === 5);
console.assert(text.charLength() === 4);
console.assert(text.toString() === "café");

Performance characteristics

Structural properties of the representations:

• Values of at most 62 bytes construct inline with no backing allocation.
• Every handle is 64 bytes; Option<Bytes> is the same size as Bytes.
• Cloning an inline value copies 64 bytes; cloning a heap-backed immutable value bumps a reference count; cloning BytesMut/Utf8BytesMut copies contents.
• Shared heap-backed conversions to and from bytes::Bytes reuse the backing allocation.
• Compact storage copies at most 62 bytes when inlining a shrinking view.

Run the benchmark suite when investigating a change:

cargo bench
cargo bench --bench clone
cargo bench --bench split_to

Development

These commands match the main CI checks:

cargo fmt --all -- --check
cargo clippy --workspace --no-default-features --features std,alloc,serde,schemars,borsh,arbitrary,quickcheck --all-targets -- -D warnings
cargo test --workspace --no-default-features --features std,serde,schemars,borsh,arbitrary,quickcheck
cargo test --package smol-bytes --no-default-features --features alloc,quickcheck
cargo check --package smol-bytes --lib --no-default-features
cargo doc --package smol-bytes --no-deps

See CONTRIBUTING.md for contribution guidance.

License

smol-bytes is available under either the MIT license or the Apache License, Version 2.0, at your option. See LICENSE-MIT and LICENSE-APACHE.

Copyright (c) 2026 Al Liu.