Crate smol_bytes 

smol-bytes

High-performance, clone-efficient byte buffers optimized for small data.

Introduction

smol-bytes provides byte buffers that store up to 62 bytes inline — no heap allocation, and cloning an inline value is a plain 64-byte copy. Larger values use bytes-backed heap storage with reference-counted clones. Two strategies control what happens when a heap value shrinks back under the inline threshold, and UTF-8 wrappers layer String-like, boundary-checked APIs over the same storage.

This is a good fit when most values are small and cloned often — tokens in a lexer, keys and field names, protocol headers — and allocation pressure matters.

Install

[dependencies]
smol-bytes = "0.2"

Python (pip install smol-bytes) and JavaScript (npm install smol-bytes) packages are also published; build-from-source instructions for each binding are below.

Quick start

The root Bytes type is an alias for shared::Bytes. Ordinary copies and static values of at most 62 bytes start inline; imported or owner-backed values and retained shared heap views can remain heap-backed below that threshold.

use smol_bytes::{Bytes, Utf8Bytes};

let small = Bytes::from_static(b"identifier");
assert!(small.is_inline());

let cloned = small.clone(); // 64-byte copy, no allocation
assert_eq!(small, cloned);

let heap = Bytes::copy_from_slice(&[0_u8; 63]);
assert!(heap.is_heap());
assert_eq!(heap.len(), 63);

let text: Utf8Bytes = "identifier".parse().expect("infallible parse");
assert_eq!(text.as_str(), "identifier");

Storage strategies

shared::Bytes preserves heap storage once a value lives there, keeping conversions with bytes::Bytes zero-copy. compact::Bytes copies a shrinking view of 62 bytes or fewer back into inline storage to release the allocation.

use smol_bytes::{compact, shared, Buf};

let mut shared = shared::Bytes::from(vec![0_u8; 100]);
let mut compact = compact::Bytes::from(vec![0_u8; 100]);

shared.advance(70);
compact.advance(70);

assert_eq!(shared.len(), 30);
assert_eq!(compact.len(), 30);
assert!(shared.is_heap());   // stays shareable and zero-copy convertible
assert!(compact.is_inline()); // allocation released, contents inlined

Rule of thumb: use shared (the default) for I/O and bytes interop; use compact when memory footprint matters more than conversion speed.

Types

Type	Storage	Mutable	Purpose
Buffer	Fixed inline bytes, up to 62 bytes	Yes	no_std fixed buffer
Bytes / shared::Bytes	Inline or shared heap-backed bytes	No	Immutable shared view
compact::Bytes	Inline or compacting heap-backed bytes	No	Immutable compacting view
BytesMut	Inline, then growable bytes::BytesMut storage	Yes	Mutable byte buffer
Utf8Buffer	Fixed inline, valid UTF-8 bytes	Yes	Small mutable UTF-8 value
Utf8Bytes / compact::Utf8Bytes	Shared or compacting UTF-8 bytes	No	Immutable UTF-8 value
Utf8BytesMut	Inline or growable valid UTF-8 bytes	Yes	Mutable UTF-8 value

Every handle is 64 bytes. With the bytes dependency enabled, all byte types implement bytes::Buf, and the mutable ones implement bytes::BufMut.

The immutable strategy types have immutable payloads but mutable views: advance, truncate, and clear change only the visible cursor/window. Utf8Bytes offers the same byte-indexed, UTF-8-boundary-checked view operations and can reclaim unique full heap allocations with try_into_mut. Utf8Buffer and Utf8BytesMut expose as_mut_str, AsMut<str>, BorrowMut<str>, DerefMut<Target = str>, and mutable string-range indexing; the standard str rules preserve UTF-8 validity.

BytesMut::split_to, split_off, and split always return BytesMut. Inline results stay inline; splitting an inline value’s reserved-but-unwritten capacity promotes it to heap storage so capacity partitioning remains exact. try_split_to and try_split_off return Result<BytesMut, OutOfBounds> and leave the receiver unchanged on bounds failure. unsplit always consumes and appends its argument, using the zero-copy bytes::BytesMut path for contiguous heap halves and copying/promoting only when required.

Rust UTF-8 split and slice indices are byte offsets that must fall on character boundaries; offenders panic, and the try_split_to, try_split_off, and try_slice variants return errors instead. (The Python bindings differ deliberately — see below.)

The UTF-8 wrappers implement FromStr. Heap-capable wrappers parse infallibly; Utf8Buffer returns TryPutError when text exceeds its fixed inline capacity.

0.2 API migration

0.2 intentionally removes several ambiguous or lossy APIs. For complete Rust, Python, and JavaScript examples, see the 0.2.0 changelog and migration guide.

• Replace Utf8Buffer::from(text) with Utf8Buffer::try_from(text)? (or Utf8Buffer::try_from_str(text)?). FromStr remains available. This makes the 62-byte fixed-capacity limit explicit.
• BytesMut split methods now return BytesMut directly. Remove an inner match/unwrap around split, split_to, and split_off; use the corresponding try_split_* method when bounds must be reported instead of panicking. try_split is removed, and unsplit now returns ().
• Use Buffer::resize(new_len, value) and Buffer::try_resize(new_len, value) to choose the byte written during extension. Failed try_resize calls are atomic.
• UTF-8 index operations now use Utf8IndexError, preserving whether an invalid character boundary, index, or full range caused the failure. Converting owned Buffer, BytesMut, or raw strategy bytes into a UTF-8 wrapper returns FromUtf8Error<B> on invalid input; call into_bytes() to recover the original allocation and view.
• Variable-width inherent Buffer::try_get_uint{,_le,_ne} and Buffer::try_get_int{,_le,_ne} methods now return TryGetIntegerError, distinguishing an invalid width from missing bytes. The bytes::Buf trait methods retain bytes’ own error type and panic contract.
• Utf8Buf and Utf8BufMut are sealed implementation traits. Generic mutable code can use try_push/try_push_str and each type’s CapacityError; Utf8Buffer reports TryPutError, while Utf8BytesMut uses Infallible.

bytes interop

Conversions with the bytes crate are zero-copy wherever the representation allows:

• bytes::Bytes -> shared::Bytes shares the allocation (From impl).
• shared::Bytes -> bytes::Bytes reuses heap backing; inline values copy.
• compact::Bytes::from(bytes::Bytes) inlines payloads of at most 62 bytes and shares larger ones.
• BytesMut::freeze_shared / freeze_compact convert without copying heap contents; BytesMut::freeze is the root-default shared conversion.
• Utf8BytesMut::freeze similarly returns the root-default Utf8Bytes; freeze_shared and freeze_compact remain available for explicit targets.
• Consuming shared::Bytes/compact::Bytes and UTF-8 strategy conversions apply the target strategy: large heap values keep their allocation and small values compact when converted to the compact strategy.
• Bytes::try_into_mut and Utf8Bytes::try_into_mut reclaim unique full heap allocations; a shared or sliced view is returned unchanged on failure.

use smol_bytes::{BytesMut, Utf8Bytes, Utf8BytesMut, compact};

let bytes = BytesMut::from(&b"payload"[..]).freeze();
assert_eq!(bytes, b"payload"[..]);

let mut text = Utf8Bytes::from("header: value");
text.advance(8);
assert_eq!(text.as_str(), "value"); // only this view moved

let compact: compact::Bytes = bytes.into();
assert_eq!(compact, b"payload"[..]);

let frozen = Utf8BytesMut::from("done").freeze();
assert_eq!(frozen.as_str(), "done");

Features and MSRV

Feature	Description
std (default)	Enables alloc, bytes/std, and Serde’s std/result support
alloc	Heap-backed types and the optional bytes dependency without std
bytes	Compatibility alias for alloc
serde	Serde support
schemars	Schemars v1 schemas; UTF-8 wrappers are strings and byte wrappers serialize as array<u8>; implies alloc
borsh	Borsh support
arbitrary	arbitrary support for generated values
quickcheck	QuickCheck support; implies std
async-graphql	Bytes and String GraphQL scalars for Bytes and Utf8Bytes; implies std
sqlx	sqlx Type/Encode/Decode for Bytes and Utf8Bytes; implies std
pyo3	Python bindings; implies std
wasm	WebAssembly bindings; implies std

With no features enabled the crate is no_std and provides the fixed Buffer and Utf8Buffer types. bytes::Buf/BufMut are available once alloc, std, or the compatibility bytes feature is selected; alloc adds the heap-backed types without std.

Byte wrapper JSON schemas describe serialization as array<u8> and accept an array or string under Schemars’ deserialization contract. Buffer additionally sets maxItems for array input and maxLength for string input to INLINE_CAP. maxLength is a scalar-count bound; deserialization still enforces the actual UTF-8 byte capacity for multibyte strings. UTF-8 wrappers remain ordinary string schemas.

The sqlx bindings decode by borrowing from the row, so a value of at most 62 bytes is built inline with no heap allocation at all. That costs one thing: PostgreSQL will not lend BYTEA out in a simple (unprepared) query, so code on that path — raw_sql, or a SQL string handed straight to an Executor — has to decode the byte types as Vec<u8> and convert. query, query_as and the query! macros carry an argument list and are therefore prepared, so they are unaffected, as are the UTF-8 types and every MySQL and SQLite path.

Rust 1.95 is the library MSRV, and the bytes dependency floor is 1.10. Development-only test and benchmark dependencies can require a newer compiler, and so do both optional integrations, to different floors: async-graphql 7.2 declares Rust 1.89, and 7.2 is the floor because the 7.0 releases do not build against the 7.2 derive crate their own dependency range admits; sqlx 0.9 declares Rust 1.94, and 0.9 is the floor because the impls are written against the lifetime-free Database::ArgumentBuffer introduced there. Enabling either raises the MSRV for the whole build.

Verification

The test and CI story is deliberately heavier than the crate’s size:

• Unit, integration, doc, and property tests (proptest state-machine comparisons against Vec/String, plus quickcheck and arbitrary generators that preserve type invariants).
• Miri over the full suite under both stacked borrows and tree borrows with strict provenance and symbolic alignment checks.
• Address, leak, memory, and thread sanitizers in CI.
• Deserialization is hardened: borsh reads length-prefixed payloads in bounded chunks instead of trusting the length prefix, and serde sequence hints are capped before preallocating.

Python

Install the published package with pip install smol-bytes (Python 3.11+).

To build from a checkout instead, Python 3.11+, Rust, and maturin are required:

python -m venv .venv
source .venv/bin/activate
python -m pip install maturin pytest
maturin develop --features pyo3 --manifest-path smol-bytes-py/Cargo.toml
python -m pytest tests/python -v

The root smol_bytes module exposes canonical shared Bytes and Utf8Bytes alongside Buffer, BytesMut, Utf8Buffer, and Utf8BytesMut. smol_bytes.shared.Bytes and smol_bytes.shared.Utf8Bytes are aliases of those exact root classes (including their root __module__), while smol_bytes.compact remains a distinct compact strategy.

Raw byte classes use from_utf8() and decode_utf8() for explicit text conversion; str(raw) is always the same total representation as str(bytes(raw)). Their from_bytes() and mutable put_slice() methods accept bytes, bytearray, and readable one-dimensional C-contiguous byte memoryview values, copying the input before mutation.

The UTF-8 classes are string-like from Python: len(), indexing, and iteration use Unicode scalar values. Their explicit APIs are char_len(), byte_len(), split_to_chars(), split_off_chars(), slice_chars(), and for mutable/fixed classes truncate_chars(). remaining_bytes() and advance_bytes() are deliberately byte-oriented, as are get_* methods.

from smol_bytes import Bytes, BytesMut, Utf8Bytes

raw = Bytes.from_bytes(b"abc")
assert raw.is_inline()
assert bytes(raw) == b"abc"
assert raw.decode_utf8() == "abc"

text = Utf8Bytes.from_str("café")
assert len(text) == 4        # Unicode characters
assert text.char_len() == 4
assert text.byte_len() == 5  # UTF-8 bytes
assert str(text) == "café"
assert str(text.split_to_chars(3)) == "caf"

mutable = BytesMut.from_bytes(bytearray(b"abc"))
frozen = mutable.freeze_shared()
assert bytes(mutable) == b""  # source remains reusable
assert bytes(frozen.to_mutable()) == b"abc"

Binding behavior worth knowing:

• Methods that allocate proportionally to caller data raise MemoryError on absurd or failing requests instead of aborting the interpreter, like CPython containers.
• memoryview(...) over the shared and compact Bytes classes exports a snapshot copy, not a live view.
• Slice assignment on the mutable classes requires matching lengths, and contiguous assignments take a direct copy fast path.
• Python unsplit() uses non-consuming snapshot semantics: it appends the call-entry contents while leaving the argument unchanged. Passing the same mutable object as both operands raises RuntimeError atomically.

JavaScript / WebAssembly

Install the published package with npm install smol-bytes.

To build from a checkout instead, install Node.js 20, wasm-pack 0.13.1, and the wasm32-unknown-unknown target. wasm-pack and Cargo resolve a compatible wasm-bindgen 0.2.x release; any separately installed wasm-bindgen helper must match the resolved crate version. CI checks the supported 0.2.x family.

rustup target add wasm32-unknown-unknown
cd js
npm ci
npm run build
npm test

The Wasm build entry point is:

wasm-pack build smol-bytes-wasm --target bundler --out-dir ../js/pkg --out-name smol_bytes -- --features wasm

The root export canonically contains Bytes and Utf8Bytes using the shared strategy, alongside Buffer, BytesMut, Utf8Buffer, and Utf8BytesMut. smol-bytes/shared exports the exact same immutable class objects; the smol-bytes/compact entry point exports distinct compact classes. Every entry point initializes iteration, disposal, and input validation independently.

Raw bytes use fromUtf8() and decodeUtf8(); they deliberately have no string-like fromString() or raw toString() API. Neither raw nor UTF-8 public classes expose fromStatic(); UTF-8 classes retain fromString() / toString() and make byte units explicit: byteLength(), charLength(), remainingBytes(), advanceBytes(), splitToBytes(), splitOffBytes(), sliceBytes(), and truncateBytes(). Byte offsets must land at UTF-8 scalar boundaries. Numeric arguments are checked before WebAssembly coercion, and malformed JavaScript UTF-16 strings are rejected before encoding.

Byte conversions at the Wasm boundary return copies. clone() is available on each public byte/string value class. Mutable freezeShared() / freezeCompact() drain their source into the selected immutable strategy, while immutable toMutable() returns an independent mutable value. JavaScript unsplit() snapshots and appends other without consuming it; self-aliasing deterministically doubles the value.

import { Bytes, BytesMut, Utf8Bytes } from "smol-bytes";
import { Bytes as CompactBytes } from "smol-bytes/compact";

const raw = CompactBytes.fromBytes(new Uint8Array([1, 2, 3]));
console.assert(raw.isInline());
console.assert(raw.toBytes()[2] === 3);

const shared = Bytes.fromUtf8("hello");
console.assert(shared.decodeUtf8() === "hello");

const mutable = BytesMut.fromUtf8("ready");
const frozen = mutable.freezeShared();
console.assert(mutable.isEmpty() && frozen.decodeUtf8() === "ready");

const text = Utf8Bytes.fromString("café");
console.assert(text.byteLength() === 5);
console.assert(text.charLength() === 4);
console.assert(text.toString() === "café");

Performance characteristics

Structural properties of the representations:

• Values of at most 62 bytes construct inline with no backing allocation.
• Every handle is 64 bytes; Option<Bytes> is the same size as Bytes.
• Cloning an inline value copies 64 bytes; cloning a heap-backed immutable value bumps a reference count; cloning BytesMut/Utf8BytesMut copies contents.
• Shared heap-backed conversions to and from bytes::Bytes reuse the backing allocation.
• Compact storage copies at most 62 bytes when inlining a shrinking view.

Run the benchmark suite when investigating a change:

cargo bench
cargo bench --bench clone
cargo bench --bench split_to

Development

These commands match the main CI checks:

cargo fmt --all -- --check
cargo clippy --workspace --no-default-features --features std,alloc,serde,schemars,borsh,arbitrary,quickcheck --all-targets -- -D warnings
cargo test --workspace --no-default-features --features std,serde,schemars,borsh,arbitrary,quickcheck
cargo test --package smol-bytes --no-default-features --features alloc,quickcheck
cargo check --package smol-bytes --lib --no-default-features
cargo doc --package smol-bytes --no-deps

See CONTRIBUTING.md for contribution guidance.

License

smol-bytes is available under either the MIT license or the Apache License, Version 2.0, at your option. See LICENSE-MIT and LICENSE-APACHE.

Copyright (c) 2026 Al Liu.

Re-exports

pub use error::*;

Modules

bufalloc or std

Utilities for working with buffers.

compactalloc or std

The Compact strategy: Aggressively inlines data to minimize memory usage.

error

Error types for byte buffer operations.

selfalloc or std

The Shared strategy: Preserves heap allocations for fast Bytes conversions.

Macros

format_compact_utf8bytesalloc or std

Formats arguments into compact Utf8Bytes.

format_utf8bytesalloc or std

Formats arguments into the default shared Utf8Bytes.

Structs

Buffer

A fixed-size buffer for inline storage.

BytesMutalloc or std

Growable mutable byte buffer with inline/heap storage.

Utf8Buffer

UTF-8 validated wrapper around Buffer with a String-like interface.

Utf8BytesMutalloc or std

Growable mutable UTF-8 string with inline/heap storage.

Utf8Drain

A borrowing iterator over a UTF-8 range removed when the iterator drops.

Constants

INLINE_CAP

Number of bytes that can be stored inline.

Traits

Bufalloc or std

Read bytes from a buffer.

BufMutalloc or std

A trait for values that provide sequential write access to bytes.

StrExtalloc or std

str methods that return default shared or compact UTF-8 bytes.

ToUtf8Bytesalloc or std

Converts a value to shared or compact UTF-8 bytes using its fmt::Display implementation.

Utf8Buf

Extension trait for UTF-8 validated buffer types.

Utf8BufMut

Extension trait for mutable UTF-8 validated buffer types.

Type Aliases

Bytesalloc or std

A space-efficient byte buffer that shares heap allocations with bytes::Bytes.

Utf8Bytesalloc or std

A shared, immutable UTF-8 string type alias using the Shared strategy.