xutf

Fast, permissive UTF-8 / UTF-16 / UTF-32 transcoding, comparison and BOM detection — plus allocation-free terminal text primitives (graphemes, visible width, truncation, word wrap).

Decoding never fails: truncated sequences, lone surrogates, garbage in → defined output out. Safe to point at untrusted bytes.

Why it's fast

The core is a Rust port of the branchless UTF kernel from xstd. The trick: UTF-8 is just a bit permutation, so one mask table drives both directions — pext gathers payload bits on decode, pdep scatters them on encode. No shuffle tables, no per-codepoint branches. Bulk paths use core::simd plus hand-written AVX-512 and NEON kernels.

The result: beats simdutf across nearly the entire grid, ASCII included, roughly 1.5–2× ahead on typical text, and 3–10× ahead on emoji-heavy input.

Features

• Every encoding pair — UTF-8/16/32, with foreign endianness as a type parameter (Utf16Be, Utf32Le, …); the byte swap is folded into the kernels.
• Cross-encoding comparison — equality, ordering, and ASCII case-folding between different encodings, no conversion needed.
• Terminal text on UTF-8 or UTF-16 — UAX #29 grapheme clusters that carry their cell width, UAX #11 width, cluster-safe truncate and word wrap. Everything returns borrowed slices.
• BOM detection — decode whatever the BOM says, defaulting to UTF-8.
• #![no_std], zero runtime dependencies, Unicode 17.0 tables generated from the UCD.
• Differentially tested — fuzzed against unicode-segmentation, exhaustive per-scalar parity with unicode-width.

Benchmarks

~1 MiB inputs, best of 7 samples, GB/s of source bytes, both hosts in each figure. simdutf's validation-skipping convert_valid_* entry points are excluded — different question. Fastest per row is bold in the raw tables; regenerate with scripts/bench_hosts.sh + scripts/bench_viz.py.

Apple M4 Max

Grapheme clusters (GB/s)

UTF-8 -> UTF-16 (GB/s)

UTF-16 -> UTF-8 (GB/s)

UTF-8 -> UTF-32 (GB/s)

UTF-32 -> UTF-8 (GB/s)

equality (GB/s)

Width truncation (GB/s)

Visible width (GB/s)

word wrap at 80 columns (GB/s)

Xeon x86-64

Grapheme clusters (GB/s)

UTF-8 -> UTF-16 (GB/s)

UTF-16 -> UTF-8 (GB/s)

UTF-8 -> UTF-32 (GB/s)

UTF-32 -> UTF-8 (GB/s)

equality (GB/s)

Width truncation (GB/s)

Visible width (GB/s)

word wrap at 80 columns (GB/s)

Highlights: graphemes 3.3–12.6× over unicode-segmentation (width included for free), wrap 6.4–27× over textwrap, fixed-width truncation O(cells) not O(bytes) (~8 ns to cut 80 cells off a 1 MiB line), equality at memcmp speed, and UTF-16 handled directly — no scratch String, no garbage.

Quick start

Requires nightly (core::simd). For peak throughput compile with RUSTFLAGS="-C target-cpu=native".

[dependencies]
xutf = { git = "https://github.com/can1357/xutf" }

use xutf::{Utf8, Utf16, Utf16Be, AsciiCase};

// Transcode between any pair of encodings (native or byte-swapped).
let utf16: Vec<u16> = xutf::transcode::<Utf8, Utf16>("naïve café 👋".as_bytes());
let be:    Vec<u16> = xutf::transcode::<Utf16, Utf16Be>(&utf16);

// Size a buffer without transcoding, or fill a fixed one.
let len = xutf::transcoded_len::<Utf16, Utf8>(&utf16);
let mut buf = vec![0u8; len];
let (read, written) = xutf::transcode_into::<Utf16, Utf8>(&utf16, &mut buf, AsciiCase::Preserve);

// Compare across encodings without converting.
assert!(xutf::equals::<Utf8, Utf16>("naïve café 👋".as_bytes(), &utf16));

// Decode whatever the BOM says (defaulting to UTF-8).
let text: Vec<u16> = xutf::from_bytes::<Utf16>(&std::fs::read("file.txt")?);

Terminal text, allocation-free and generic over the encoding:

let line = "naïve café 界面 👨‍👩‍👧 done";
assert_eq!(xutf::width_str(line), 25);
let cut = xutf::truncate_str(line, 12);            // borrowed prefix, no alloc
let cols: usize = xutf::graphemes_str(line).count();
let rows = xutf::wrap_str(line, 10).count();

Per-character widths match unicode-width 0.2.2 exactly (sole exception U+17D8); string widths are cluster-exact — width(s) always equals the sum of graphemes(s) item widths. Deliberate divergences from the unicode-width string automaton are listed in src/width.rs.

Fuzzing

Two libFuzzer targets check the public interface against slower reference implementations, over arbitrary byte soup reinterpreted as u8 / u16 / u32 units in every encoding (native and byte-swapped):

cargo fuzz run codec -- -max_total_time=60   # codecs, transcode, compare, BOM
cargo fuzz run text  -- -max_total_time=60   # graphemes, width, truncate, wrap

codec pins every encoding pair against scalar decode/encode: unit counts (transcoded_len = transcode), bounded transcode_into prefixes, codepoint ordering, and BOM dispatch. text pins cluster boundaries against unicode-segmentation on valid text, and on malformed input pins the allocation-free primitives to the input's own cluster decomposition: widths sum per cluster, truncation and wrapping only cut on boundaries, and wrapping drops nothing but space and line-break separators.

License

MIT