Crate xutf 

Permissive UTF-8 / UTF-16 / UTF-32 transcoding, comparison and BOM detection, with SIMD ASCII fast paths.

A Rust port of the xstd::codepoint_cvt / utf_* family. Design points carried over:

• Permissive, non-validating codecs. Decoding never fails: truncated UTF-8 sequences decode to 0 (consuming the remainder), lone surrogates pass through, garbage-in produces garbage-out. Values are raw u32 codepoints, not char.
• ASCII-only case folding for the *_ignore_ascii_case operations and AsciiCase transcoding.
• Foreign endianness as a type parameter: Utf16<true> is byte-swapped relative to the native byte order (see Utf16Be / Utf16Le aliases).
• Terminal text primitives, allocation-free and generic over the encoding: grapheme segmentation (graphemes, UAX #29 incl. GB9c and GB11), visible cell width (width), width truncation (truncate, returns a borrowed prefix) and greedy word wrap (wrap, yields borrowed sublines), backed by a generated Unicode UNICODE_VERSION property trie (scripts/gen_props.py).

Special-op mapping from the C++ original:

C++ (xstd)	Rust
clang vector extensions (xvec)	core::simd (LLVM vector IR)
bit_pdep / bit_pext (BMI2)	_pdep_u32 / _pext_u32 under cfg(bmi2)
assume(..)	core::hint::assert_unchecked
bswapw / bswapd	u16::swap_bytes / u32::swap_bytes
lsb(mask())	Mask::to_bitmask().trailing_zeros()
overlapped tail vector load/store	Simd::from_slice / copy_to_slice on an
	overlapping window at limit - N

Deliberate fixes over the original:

• UTF-16 encode computes the low surrogate as 0xDC00 | (cp & 0x3FF); the C++ 0xDC00 | uint16_t(cp) corrupts pairs when bit 13 of cp - 0x10000 is set (e.g. U+12000).
• compare orders by decoded codepoint everywhere (the C++ mixed UTF-16 code-unit order with codepoint order) and a longer string with a trailing NUL codepoint no longer compares equal to its prefix.

Structs

Codepoints

Iterator over the raw codepoints of an encoded slice.

Grapheme

A borrowed extended grapheme cluster and its terminal cell width.

Graphemes

Allocation-free iterator over extended grapheme clusters.

Utf8

UTF-8. Byte-oriented, so there is no foreign-endianness variant.

Utf16

UTF-16. FOREIGN selects byte order opposite to native (see Utf16Le / Utf16Be).

Utf32

UTF-32. FOREIGN selects byte order opposite to native (see Utf32Le / Utf32Be).

Wrapped

Iterator of wrapped lines, created by wrap.

Enums

AsciiCase

ASCII case transform applied while transcoding.

Bom

Byte order mark at the head of a raw byte stream.

Kind

Encoding family, used to detect “same encoding” fast paths at monomorphization time.

Constants

UNICODE_VERSION

Unicode version of the generated property tables.

Traits

Encoding

A UTF encoding: a stateless codec between raw u32 codepoints and code units.

Unit

A UTF code unit: u8 (UTF-8), u16 (UTF-16) or u32 (UTF-32).

Functions

chars

Iterates input as chars, substituting U+FFFD for unit sequences that decode to invalid scalar values.

codepoints

Iterates the raw (unvalidated) codepoints of input.

compare

Compares two encoded strings in codepoint order.

compare_ignore_ascii_case

compare with ASCII-only case folding (utf_icompare).

detect_bom

Detects a BOM, returning it plus the byte length to skip.

equals

Codepoint equality across encodings (utf_cmpeq).

equals_ignore_ascii_case

equals with ASCII-only case folding (utf_icmpeq).

from_bytes

Identifies the encoding of a raw byte stream from its BOM (defaulting to UTF-8) and transcodes it to T.

graphemes

Iterates the extended grapheme clusters of an encoded slice without allocating.

graphemes_str

Iterates the extended grapheme clusters of a UTF-8 string as borrowed strings.

to_string

Transcodes to an owned String.

transcode

Transcodes all of src into a freshly allocated unit vector.

transcode_into

Transcodes as much of src into dst as fits, returning (units_read, units_written). Never splits a codepoint: encoding stops at the last codepoint whose output fits.

transcode_with_case

transcode with ASCII case folding.

transcoded_len

Unit count src would occupy once transcoded to T (utf_length).

truncate

Longest prefix of input no wider than max_width terminal cells, cut on an extended grapheme cluster boundary.

truncate_str

truncate over UTF-8 str.

width

Visible width of input in terminal cells (extended grapheme clusters, UAX #11 plus emoji presentation rules, with an ASCII bulk path).

width_str

width over UTF-8 str.

wrap

Greedily wraps input into at least one borrowed terminal-width subline.

wrap_str

Applies wrap to a UTF-8 string.

Type Aliases

Utf16Be

Big-endian UTF-16, regardless of the native byte order.

Utf16Le

Little-endian UTF-16, regardless of the native byte order.

Utf32Be

Big-endian UTF-32, regardless of the native byte order.

Utf32Le

Little-endian UTF-32, regardless of the native byte order.