# xutf
[](https://github.com/can1357/xutf/actions/workflows/ci.yml)
[](#license)
Fast, permissive UTF-8 / UTF-16 / UTF-32 transcoding, comparison and BOM
detection — plus allocation-free terminal text primitives (graphemes, visible
width, truncation, word wrap).
Decoding never fails: truncated sequences, lone surrogates, garbage in →
defined output out. Safe to point at untrusted bytes.
## Why it's fast
The core is a Rust port of the branchless UTF kernel from
[`xstd`](https://github.com/can1357/xstd/blob/master/includes/xstd/utf.hpp).
The trick: UTF-8 is just a bit permutation, so one mask table drives both
directions — `pext` gathers payload bits on decode, `pdep` scatters them on
encode. No shuffle tables, no per-codepoint branches. Bulk paths use
`core::simd` plus hand-written AVX-512 and NEON kernels.
The result: beats `simdutf` across nearly the entire grid, ASCII included,
roughly 1.5–2× ahead on typical text, and **3–10×** ahead on emoji-heavy input.
## Features
- **Every encoding pair** — UTF-8/16/32, with foreign endianness as a type
parameter (`Utf16Be`, `Utf32Le`, …); the byte swap is folded into the kernels.
- **Cross-encoding comparison** — equality, ordering, and ASCII case-folding
between different encodings, no conversion needed.
- **Terminal text on UTF-8 _or_ UTF-16** — UAX #29 grapheme clusters that
carry their cell width, UAX #11 width, cluster-safe truncate and word wrap.
Everything returns borrowed slices.
- **BOM detection** — decode whatever the BOM says, defaulting to UTF-8.
- **`#![no_std]`**, zero runtime dependencies, Unicode 17.0 tables generated
from the UCD.
- **Differentially tested** — fuzzed against `unicode-segmentation`,
exhaustive per-scalar parity with `unicode-width`.
## Benchmarks
~1 MiB inputs, best of 7 samples, GB/s of source bytes, both hosts in each
figure. `simdutf`'s validation-skipping `convert_valid_*` entry points are
excluded — different question. Fastest per row is **bold** in the raw
tables; regenerate with `scripts/bench_hosts.sh` + `scripts/bench_viz.py`.
<details><summary>Raw numbers per host</summary>
### Apple M4 Max
**Grapheme clusters** (GB/s)
| ascii | **1.9 GB/s (1.00x)** | 0.1 GB/s (▼ +1221.9%) | 0.1 GB/s (▼ +1403.0%) | 1.6 GB/s (▼ +18.1%) | 0.1 GB/s (▼ +1546.0%) |
| cjk | 0.8 GB/s (1.00x) | 0.2 GB/s (▼ +253.0%) | 0.2 GB/s (▼ +305.9%) | **1.1 GB/s (▲ -31.1%)** | 0.2 GB/s (▼ +333.9%) |
| emoji-soup | 0.8 GB/s (1.00x) | 0.2 GB/s (▼ +249.7%) | 0.2 GB/s (▼ +340.4%) | **1.0 GB/s (▲ -30.1%)** | 0.2 GB/s (▼ +368.9%) |
| mixed | 0.8 GB/s (1.00x) | 0.2 GB/s (▼ +382.8%) | 0.1 GB/s (▼ +456.7%) | **1.0 GB/s (▲ -22.7%)** | 0.1 GB/s (▼ +513.9%) |
**UTF-8 -> UTF-16** (GB/s)
| ascii | **48.7 GB/s (1.00x)** | 41.2 GB/s (▼ +18.2%) | 6.9 GB/s (▼ +605.9%) | 2.0 GB/s (▼ +2378.0%) |
| cjk-3b | **9.7 GB/s (1.00x)** | 8.0 GB/s (▼ +20.9%) | 4.2 GB/s (▼ +132.7%) | 2.8 GB/s (▼ +241.2%) |
| emoji-4b | **5.5 GB/s (1.00x)** | 5.1 GB/s (▼ +7.3%) | 3.1 GB/s (▼ +77.9%) | 3.1 GB/s (▼ +78.3%) |
**UTF-16 -> UTF-8** (GB/s)
| ascii | **56.0 GB/s (1.00x)** | 41.4 GB/s (▼ +35.3%) | 17.9 GB/s (▼ +213.8%) | 3.7 GB/s (▼ +1411.2%) |
| cjk-3b | **9.5 GB/s (1.00x)** | **9.5 GB/s (1.00x)** | 2.4 GB/s (▼ +301.7%) | 1.7 GB/s (▼ +457.2%) |
| emoji-4b | **10.7 GB/s (1.00x)** | 3.5 GB/s (▼ +208.8%) | 3.1 GB/s (▼ +243.1%) | 2.9 GB/s (▼ +265.0%) |
**UTF-8 -> UTF-32** (GB/s)
| ascii | 17.1 GB/s (1.00x) | **17.3 GB/s (▲ -1.1%)** | 2.0 GB/s (▼ +776.4%) |
| cjk-3b | **8.9 GB/s (1.00x)** | 7.6 GB/s (▼ +18.0%) | 3.7 GB/s (▼ +141.2%) |
| emoji-4b | **7.0 GB/s (1.00x)** | 6.8 GB/s (▼ +2.8%) | 5.3 GB/s (▼ +32.6%) |
**UTF-32 -> UTF-8** (GB/s)
| ascii | **66.1 GB/s (1.00x)** | 61.8 GB/s (▼ +7.0%) | 7.8 GB/s (▼ +750.1%) |
| cjk-3b | **18.8 GB/s (1.00x)** | 17.5 GB/s (▼ +7.8%) | 4.2 GB/s (▼ +350.7%) |
| emoji-4b | **18.0 GB/s (1.00x)** | 4.6 GB/s (▼ +294.0%) | 3.8 GB/s (▼ +378.1%) |
**equality** (GB/s)
| ascii | **57.3 GB/s (1.00x)** | 55.8 GB/s (▼ +2.6%) | 20.6 GB/s (▼ +178.4%) | 20.7 GB/s (▼ +176.1%) | 13.1 GB/s (▼ +337.0%) | 5.5 GB/s (▼ +934.1%) |
| cjk-3b | **57.3 GB/s (1.00x)** | 56.0 GB/s (▼ +2.2%) | 20.6 GB/s (▼ +178.3%) | 20.8 GB/s (▼ +175.6%) | 3.4 GB/s (▼ +1563.6%) | 4.0 GB/s (▼ +1338.4%) |
| emoji-4b | **57.4 GB/s (1.00x)** | 56.0 GB/s (▼ +2.4%) | 20.6 GB/s (▼ +179.2%) | 20.7 GB/s (▼ +177.0%) | 1.7 GB/s (▼ +3311.9%) | 2.9 GB/s (▼ +1895.2%) |
**Width truncation** (GB/s)
| ascii/80 | **124901.7 GB/s (1.00x)** | 1263.0 GB/s (▼ +9789.3%) | 959.9 GB/s (▼ +12912.5%) |
| ascii/half | **116.7 GB/s (1.00x)** | 0.2 GB/s (▼ +59202.3%) | 0.2 GB/s (▼ +64668.3%) |
| cjk/80 | **5849.5 GB/s (1.00x)** | 1637.1 GB/s (▼ +257.3%) | 1253.8 GB/s (▼ +366.5%) |
| cjk/half | **1.3 GB/s (1.00x)** | 0.4 GB/s (▼ +261.4%) | 0.3 GB/s (▼ +302.2%) |
| emoji/80 | **6670.1 GB/s (1.00x)** | 854.5 GB/s (▼ +680.6%) | 725.4 GB/s (▼ +819.5%) |
| emoji/half | **2.0 GB/s (1.00x)** | 0.2 GB/s (▼ +718.4%) | 0.2 GB/s (▼ +779.8%) |
**Visible width** (GB/s)
| ascii | **60.1 GB/s (1.00x)** | 1.6 GB/s (▼ +3572.6%) | 28.5 GB/s (▼ +110.7%) | 0.7 GB/s (▼ +8155.4%) |
| cjk | **1.6 GB/s (1.00x)** | 1.1 GB/s (▼ +44.1%) | 1.5 GB/s (▼ +9.7%) | 0.8 GB/s (▼ +115.9%) |
| emoji | **1.0 GB/s (1.00x)** | 0.9 GB/s (▼ +14.7%) | 1.0 GB/s (▼ +2.4%) | 0.6 GB/s (▼ +68.4%) |
| mixed | **1.5 GB/s (1.00x)** | 1.1 GB/s (▼ +36.3%) | 1.3 GB/s (▼ +20.2%) | 0.6 GB/s (▼ +140.0%) |
**word wrap at 80 columns** (GB/s)
| ascii | 0.6 GB/s (1.00x) | 0.0 GB/s (▼ +1319.5%) | **1.1 GB/s (▲ -100.1%)** |
| cjk | **0.6 GB/s (1.00x)** | 0.0 GB/s (▼ +2150.7%) | 0.4 GB/s (▼ +42.0%) |
| mixed-emoji | 0.5 GB/s (1.00x) | 0.0 GB/s (▼ +997.0%) | **0.9 GB/s (▲ -62.9%)** |
### Xeon x86-64
**Grapheme clusters** (GB/s)
| ascii | **1.1 GB/s (1.00x)** | 0.1 GB/s (▼ +1166.3%) | 0.1 GB/s (▼ +1354.1%) | **1.1 GB/s (1.00x)** | 0.1 GB/s (▼ +1470.3%) |
| cjk | 0.7 GB/s (1.00x) | 0.2 GB/s (▼ +317.8%) | 0.1 GB/s (▼ +406.1%) | **0.7 GB/s (▲ -6.6%)** | 0.1 GB/s (▼ +428.5%) |
| emoji-soup | 0.7 GB/s (1.00x) | 0.2 GB/s (▼ +324.7%) | 0.1 GB/s (▼ +417.4%) | **0.7 GB/s (▲ -1.3%)** | 0.1 GB/s (▼ +423.6%) |
| mixed | 0.7 GB/s (1.00x) | 0.1 GB/s (▼ +431.2%) | 0.1 GB/s (▼ +521.4%) | **0.7 GB/s (▲ -1.4%)** | 0.1 GB/s (▼ +566.8%) |
**UTF-8 -> UTF-16** (GB/s)
| ascii | 25.6 GB/s (1.00x) | 22.2 GB/s (▼ +15.5%) | **27.2 GB/s (▲ -6.3%)** | 2.1 GB/s (▼ +1125.8%) |
| cjk-3b | **8.1 GB/s (1.00x)** | 6.7 GB/s (▼ +20.1%) | 2.6 GB/s (▼ +211.5%) | 2.6 GB/s (▼ +206.1%) |
| emoji-4b | **6.6 GB/s (1.00x)** | 4.6 GB/s (▼ +42.2%) | 2.3 GB/s (▼ +191.1%) | 2.4 GB/s (▼ +171.6%) |
**UTF-16 -> UTF-8** (GB/s)
| ascii | **72.0 GB/s (1.00x)** | **72.7 GB/s (1.00x)** | 32.4 GB/s (▼ +122.3%) | 2.5 GB/s (▼ +2835.5%) |
| cjk-3b | **11.5 GB/s (1.00x)** | 8.3 GB/s (▼ +38.7%) | 1.7 GB/s (▼ +577.4%) | 1.2 GB/s (▼ +824.2%) |
| emoji-4b | **8.7 GB/s (1.00x)** | 6.8 GB/s (▼ +29.1%) | 2.4 GB/s (▼ +267.6%) | 1.7 GB/s (▼ +421.5%) |
**UTF-8 -> UTF-32** (GB/s)
| ascii | **10.2 GB/s (1.00x)** | 9.9 GB/s (▼ +2.8%) | 3.0 GB/s (▼ +243.6%) |
| cjk-3b | **7.9 GB/s (1.00x)** | 5.3 GB/s (▼ +48.3%) | 3.6 GB/s (▼ +122.2%) |
| emoji-4b | **8.7 GB/s (1.00x)** | 5.4 GB/s (▼ +61.3%) | 4.2 GB/s (▼ +106.1%) |
**UTF-32 -> UTF-8** (GB/s)
| ascii | **83.1 GB/s (1.00x)** | 52.7 GB/s (▼ +57.9%) | 5.7 GB/s (▼ +1354.6%) |
| cjk-3b | **20.4 GB/s (1.00x)** | 13.9 GB/s (▼ +46.4%) | 3.0 GB/s (▼ +575.2%) |
| emoji-4b | **30.5 GB/s (1.00x)** | 3.2 GB/s (▼ +853.9%) | 3.5 GB/s (▼ +767.8%) |
**equality** (GB/s)
| ascii | **82.2 GB/s (1.00x)** | **82.2 GB/s (1.00x)** | 16.4 GB/s (▼ +400.4%) | 17.3 GB/s (▼ +375.4%) | 12.7 GB/s (▼ +548.9%) | 14.0 GB/s (▼ +488.3%) |
| cjk-3b | **82.0 GB/s (1.00x)** | **82.3 GB/s (1.00x)** | 16.4 GB/s (▼ +401.4%) | 17.3 GB/s (▼ +374.5%) | 3.0 GB/s (▼ +2622.1%) | 2.7 GB/s (▼ +2929.5%) |
| emoji-4b | **81.9 GB/s (1.00x)** | **82.1 GB/s (1.00x)** | 17.1 GB/s (▼ +377.7%) | 18.1 GB/s (▼ +352.9%) | 1.5 GB/s (▼ +5223.7%) | 2.1 GB/s (▼ +3712.1%) |
**Width truncation** (GB/s)
| ascii/80 | **130401.8 GB/s (1.00x)** | 876.4 GB/s (▼ +14779.1%) | 734.7 GB/s (▼ +17648.7%) |
| ascii/half | **180.7 GB/s (1.00x)** | 0.1 GB/s (▼ +136719.9%) | 0.1 GB/s (▼ +147691.6%) |
| cjk/80 | **4507.6 GB/s (1.00x)** | 1100.1 GB/s (▼ +309.7%) | 939.1 GB/s (▼ +380.0%) |
| cjk/half | **1.0 GB/s (1.00x)** | 0.3 GB/s (▼ +306.9%) | 0.2 GB/s (▼ +329.4%) |
| emoji/80 | **4576.1 GB/s (1.00x)** | 647.3 GB/s (▼ +607.0%) | 558.7 GB/s (▼ +719.0%) |
| emoji/half | **1.5 GB/s (1.00x)** | 0.2 GB/s (▼ +731.2%) | 0.2 GB/s (▼ +746.3%) |
**Visible width** (GB/s)
| ascii | **92.5 GB/s (1.00x)** | 1.2 GB/s (▼ +7424.3%) | 24.9 GB/s (▼ +271.8%) | 0.7 GB/s (▼ +14023.4%) |
| cjk | 0.8 GB/s (1.00x) | **1.0 GB/s (▲ -15.4%)** | 0.9 GB/s (▲ -8.8%) | 0.6 GB/s (▼ +32.0%) |
| emoji | 0.6 GB/s (1.00x) | **0.7 GB/s (▲ -23.2%)** | 0.6 GB/s (▲ -9.0%) | 0.5 GB/s (▼ +20.1%) |
| mixed | 0.9 GB/s (1.00x) | 0.8 GB/s (▼ +6.4%) | **0.9 GB/s (▲ -1.8%)** | 0.5 GB/s (▼ +72.1%) |
**word wrap at 80 columns** (GB/s)
| ascii | 0.4 GB/s (1.00x) | 0.0 GB/s (▼ +1157.4%) | **0.8 GB/s (▲ -114.2%)** |
| cjk | **0.6 GB/s (1.00x)** | 0.0 GB/s (▼ +2788.6%) | 0.4 GB/s (▼ +41.5%) |
| mixed-emoji | 0.4 GB/s (1.00x) | 0.0 GB/s (▼ +905.9%) | **0.7 GB/s (▲ -63.9%)** |
</details>
Highlights: graphemes 3.3–12.6× over `unicode-segmentation` (width included
for free), wrap 6.4–27× over `textwrap`, fixed-width truncation O(cells) not
O(bytes) (~8 ns to cut 80 cells off a 1 MiB line), equality at `memcmp`
speed, and UTF-16 handled directly — no scratch `String`, no garbage.
## Quick start
Requires nightly (`core::simd`). For peak throughput compile with
`RUSTFLAGS="-C target-cpu=native"`.
```toml
[dependencies]
xutf = { git = "https://github.com/can1357/xutf" }
```
```rust
use xutf::{Utf8, Utf16, Utf16Be, AsciiCase};
// Transcode between any pair of encodings (native or byte-swapped).
let utf16: Vec<u16> = xutf::transcode::<Utf8, Utf16>("naïve café 👋".as_bytes());
let be: Vec<u16> = xutf::transcode::<Utf16, Utf16Be>(&utf16);
// Size a buffer without transcoding, or fill a fixed one.
let len = xutf::transcoded_len::<Utf16, Utf8>(&utf16);
let mut buf = vec![0u8; len];
let (read, written) = xutf::transcode_into::<Utf16, Utf8>(&utf16, &mut buf, AsciiCase::Preserve);
// Compare across encodings without converting.
assert!(xutf::equals::<Utf8, Utf16>("naïve café 👋".as_bytes(), &utf16));
// Decode whatever the BOM says (defaulting to UTF-8).
let text: Vec<u16> = xutf::from_bytes::<Utf16>(&std::fs::read("file.txt")?);
```
Terminal text, allocation-free and generic over the encoding:
```rust
let line = "naïve café 界面 👨👩👧 done";
assert_eq!(xutf::width_str(line), 25);
let cut = xutf::truncate_str(line, 12); // borrowed prefix, no alloc
let cols: usize = xutf::graphemes_str(line).count();
let rows = xutf::wrap_str(line, 10).count();
```
Per-character widths match `unicode-width` 0.2.2 exactly (sole exception
U+17D8); string widths are cluster-exact — `width(s)` always equals the sum
of `graphemes(s)` item widths. Deliberate divergences from the
`unicode-width` string automaton are listed in `src/width.rs`.
## Fuzzing
Two libFuzzer targets check the public interface against slower reference
implementations, over arbitrary byte soup reinterpreted as `u8` / `u16` /
`u32` units in every encoding (native and byte-swapped):
```sh
cargo fuzz run codec -- -max_total_time=60 # codecs, transcode, compare, BOM
cargo fuzz run text -- -max_total_time=60 # graphemes, width, truncate, wrap
```
`codec` pins every encoding pair against scalar decode/encode: unit counts
(`transcoded_len` = `transcode`), bounded `transcode_into` prefixes, codepoint
ordering, and BOM dispatch. `text` pins cluster boundaries against
`unicode-segmentation` on valid text, and on malformed input pins the
allocation-free primitives to the input's own cluster decomposition: widths
sum per cluster, truncation and wrapping only cut on boundaries, and wrapping
drops nothing but space and line-break separators.
## License
MIT