Crate xutf

Expand description

Permissive UTF-8 / UTF-16 / UTF-32 transcoding, comparison and BOM detection, with SIMD ASCII fast paths.

A Rust port of the xstd::codepoint_cvt / utf_* family. Design points carried over:

Permissive, non-validating codecs. Decoding never fails: truncated UTF-8 sequences decode to 0 (consuming the remainder), lone surrogates pass through, garbage-in produces garbage-out. Values are raw u32 codepoints, not char.
ASCII-only case folding for the *_ignore_ascii_case operations and AsciiCase transcoding.
Foreign endianness as a type parameter: Utf16<true> is byte-swapped relative to the native byte order (see Utf16Be / Utf16Le aliases).
Terminal text primitives, allocation-free and generic over the encoding: grapheme segmentation (graphemes, UAX #29 incl. GB9c and GB11) with double-ended, exact-length iterators, visible cell width (width), width truncation (truncate), greedy word wrap (wrap), and SIMD ANSI/VT stripping in place (MakeAnsiStripped, IntoAnsiStripped). The Text extension trait exposes them as methods on str and native code-unit slices. Unicode properties come from the generated UNICODE_VERSION trie (scripts/gen_props.py).
Scalar property lookups: NFC/NFD normalization with a SIMD quick-check (ToUnicodeNormalized, is_nfc), plus permissive General_Category (general_category, general_category_group) and UAX #24 script (script) — the lookup surface of the unicode-normalization, unicode-properties and unicode-script crates from one table family, also as Ucd methods on char/u32.
Strict stream decoding from any std::io::BufRead (BufReadCharsExt): batch-decoded char iteration in every supported encoding, with errors that carry the offending bytes (ReadCharError). Drop-in compatible with (and considerably faster than) the utf8-chars crate.
Method-level entry points for the codec itself: Text::transcode infers the target encoding from the container it fills (TextBuf), Text::eq_text compares across encodings, and Encoding::from_bytes decodes a BOM-tagged byte stream into that encoding’s Container.

Special-op mapping from the C++ original:

C++ (`xstd`)	Rust
clang vector extensions (`xvec`)	`core::simd` (LLVM vector IR)
`bit_pdep` / `bit_pext` (BMI2)	`_pdep_u32` / `_pext_u32` under `cfg(bmi2)`
`assume(..)`	`core::hint::assert_unchecked`
`bswapw` / `bswapd`	`u16::swap_bytes` / `u32::swap_bytes`
`lsb(mask())`	`Mask::to_bitmask().trailing_zeros()`
overlapped tail vector load/store	`Simd::from_slice` / `copy_to_slice` on an
	overlapping window at `limit - N`

Deliberate fixes over the original:

UTF-16 encode computes the low surrogate as 0xDC00 | (cp & 0x3FF); the C++ 0xDC00 | uint16_t(cp) corrupts pairs when bit 13 of cp - 0x10000 is set (e.g. U+12000).
compare orders by decoded codepoint everywhere (the C++ mixed UTF-16 code-unit order with codepoint order) and a longer string with a trailing NUL codepoint no longer compares equal to its prefix.

Structs§

Chars: Iterator over the chars of a BufRead, yielding io::Result<char>.
CharsRaw: Iterator over the chars of a BufRead, yielding Result<char, ReadCharError>.
Codepoints: Iterator over the raw codepoints of an encoded slice.
Grapheme: A borrowed extended grapheme cluster and its terminal cell width.
GraphemeIndices: Allocation-free double-ended iterator over extended grapheme clusters and their code-unit offsets.
Graphemes: Allocation-free double-ended iterator over extended grapheme clusters.
NormalizationError: Reports the workspace capacity needed for allocation-free normalization.
ReadCharError: Bytes read from a stream that failed to decode, plus the io::Error classifying the failure.
StrGraphemeIndices: Double-ended, exact-length iterator over a UTF-8 string’s grapheme clusters and byte offsets.
StrGraphemes: Double-ended, exact-length iterator over the extended grapheme clusters of a UTF-8 string, yielding borrowed sub-strings. Created by graphemes_str.
StrWrapped: Iterator of wrapped &str lines, created by wrap_str.
Utf8: UTF-8. Byte-oriented, so there is no foreign-endianness variant.
Utf16: UTF-16. FOREIGN selects byte order opposite to native (see Utf16Le / Utf16Be).
Utf32: UTF-32. FOREIGN selects byte order opposite to native (see Utf32Le / Utf32Be).
Wrapped: Iterator of wrapped lines, created by wrap.
WrappedLine: A borrowed wrapped subline with its position and terminal cell width.
WrappedMeasured: Iterator of wrapped lines with source offsets and visible widths, created by wrap_measured.

Enums§

AsciiCase: ASCII case transform applied while transcoding.
Bom: Byte order mark at the head of a raw byte stream.
GeneralCategory: General_Category property value (UAX #44), e.g. \p{Lu}.
GeneralCategoryGroup: The seven General_Category groups: \p{L}, \p{M}, \p{N}, \p{P}, \p{S}, \p{Z}, \p{C}.
Kind: Encoding family, used to detect “same encoding” fast paths at monomorphization time.
Script: Script property values (UAX #24) of UCD 17.0.0.

Constants§

UNICODE_VERSION: Unicode version of the generated property tables.

Traits§

BufReadCharsExt: Extends BufRead with strict char decoding, batch-accelerated over the reader’s internal buffer.
Encoding: A UTF encoding: a stateless codec between raw u32 codepoints and code units.
IntoAnsiStripped: Converts an owned string into ANSI/VT-free text without reallocating.
IntoUnicodeNormalized: Converts an owned string to NFC or NFD while reusing its allocation.
MakeAnsiStripped: Compacts ANSI/VT-free text into a mutable code-unit slice view.
MakeUnicodeNormalized: Normalizes a string in place without allocating.
StreamDecode: An encoding whose chars can be strictly decoded from a byte stream.
Text: Terminal text primitives as methods on str and native code-unit slices.
TextBuf: Owned destination of Text::transcode; pins the target encoding through its code-unit type.
ToAnsiStripped: Creates an ANSI/VT-free owned string from a borrowed string.
ToUnicodeNormalized: Creates owned NFC or NFD text from a borrowed string.
Ucd: general_category, general_category_group and script as methods on the scalar itself.
Unit: A UTF code unit: u8 (UTF-8), u16 (UTF-16) or u32 (UTF-32).

Functions§

canonical_combining_class: Canonical Combining Class (ccc) of a codepoint; 0 for starters, out-of-range input and unassigned codepoints.
chars: Iterates input as chars, substituting U+FFFD for unit sequences that decode to invalid scalar values.
codepoints: Iterates the raw (unvalidated) codepoints of input.
compare: Compares two encoded strings in codepoint order.
compare_ignore_ascii_case: compare with ASCII-only case folding (utf_icompare).
detect_bom: Detects a BOM, returning it plus the byte length to skip.
equals: Codepoint equality across encodings (utf_cmpeq).
equals_ignore_ascii_case: equals with ASCII-only case folding (utf_icmpeq).
from_bytes: Identifies the encoding of a raw byte stream from its BOM (defaulting to UTF-8) and transcodes it to T.
general_category: General_Category of a codepoint.
general_category_group: general_category reduced to its group: the \p{L}/\p{N}/\p{M} classes of the reference tokenizer regexes in one lookup.
grapheme_indices: Iterates the extended grapheme clusters of an encoded slice with their code-unit offsets.
grapheme_indices_str: Iterates a UTF-8 string’s extended grapheme clusters with their byte offsets.
graphemes: Iterates the extended grapheme clusters of an encoded slice without allocating.
graphemes_str: Iterates the extended grapheme clusters of a UTF-8 string as borrowed strings.
is_nfc: Reports whether text is definitely in NFC form.
is_nfc_codepoints: is_nfc over raw codepoints, for callers holding non-UTF-8 text (UTF-16/UTF-32 units) who want the quick-check without transcoding.
script: Script property of a codepoint (UAX #24), e.g. \p{Han}.
skip_columns: Tail of input after dropping enough whole clusters to cover columns terminal cells, together with the exact dropped width.
skip_columns_str: skip_columns over UTF-8 str.
to_string: Transcodes to an owned String.
transcode: Transcodes all of src into a freshly allocated unit vector.
transcode_into: Transcodes as much of src into dst as fits, returning (units_read, units_written). Never splits a codepoint: encoding stops at the last codepoint whose output fits.
transcode_with_case: transcode with ASCII case folding.
transcoded_len: Unit count src would occupy once transcoded to T (utf_length).
truncate: Longest prefix of input no wider than max_width terminal cells, cut on an extended grapheme cluster boundary.
truncate_measured: Longest prefix of input no wider than max_width terminal cells, cut on an extended grapheme cluster boundary, and its exact terminal-cell width.
truncate_measured_str: truncate_measured over UTF-8 str.
truncate_str: truncate over UTF-8 str.
width: Visible width of input in terminal cells (extended grapheme clusters, UAX #11 plus emoji presentation rules, with an ASCII bulk path).
width_ansi: Visible terminal width of input, ignoring ANSI/VT escape sequences.
width_ansi_str: Visible terminal width of a UTF-8 string, ignoring ANSI/VT escape sequences.
width_char: Standalone terminal-cell width of c.
width_str: width over UTF-8 str.
width_within: Visible width of input when it is at most max_width, or None once the running width exceeds the bound.
width_within_str: width_within over UTF-8 str.
wrap: Greedily wraps input into at least one borrowed terminal-width subline.
wrap_measured: Greedily wraps input, reporting each subline’s offset and visible width.
wrap_measured_str: Applies wrap_measured to a UTF-8 string.
wrap_str: Applies wrap to a UTF-8 string.

Type Aliases§

Utf16Be: Big-endian UTF-16, regardless of the native byte order.
Utf16Le: Little-endian UTF-16, regardless of the native byte order.
Utf32Be: Big-endian UTF-32, regardless of the native byte order.
Utf32Le: Little-endian UTF-32, regardless of the native byte order.

Crate xutf

Crate xutf Copy item path

Structs§

Enums§

Constants§

Traits§

Functions§

Type Aliases§

Crate xutf