Module encoding_rs::mem

source ·
Expand description

Functions for converting between different in-RAM representations of text and for quickly checking if the Unicode Bidirectional Algorithm can be avoided.

By using slices for output, the functions here seek to enable by-register (ALU register or SIMD register as available) operations in order to outperform iterator-based conversions available in the Rust standard library.

Note: “Latin1” in this module refers to the Unicode range from U+0000 to U+00FF, inclusive, and does not refer to the windows-1252 range. This in-memory encoding is sometimes used as a storage optimization of text when UTF-16 indexing and length semantics are exposed.

The FFI binding for this module are in the encoding_c_mem crate.

Enums§

  • Classification of text as Latin1 (all code points are below U+0100), left-to-right with some non-Latin1 characters or as containing at least some right-to-left characters.

Functions§

  • Checks whether a valid UTF-8 buffer contains code points that trigger right-to-left processing or is all-Latin1.
  • Checks whether a potentially invalid UTF-8 buffer contains code points that trigger right-to-left processing or is all-Latin1.
  • Checks whether a potentially invalid UTF-16 buffer contains code points that trigger right-to-left processing or is all-Latin1.
  • Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 such that the validity of the output is signaled using the Rust type system.
  • Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 such that the validity of the output is signaled using the Rust type system with potentially insufficient output space.
  • Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8.
  • Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8 with potentially insufficient output space.
  • Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-16.
  • Converts valid UTF-8 to valid UTF-16.
  • If the input is valid UTF-8 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.
  • Converts potentially-invalid UTF-8 to valid UTF-16 with errors replaced with the REPLACEMENT CHARACTER.
  • Converts potentially-invalid UTF-8 to valid UTF-16 signaling on error.
  • If the input is valid UTF-16 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.
  • Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER such that the validity of the output is signaled using the Rust type system.
  • Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER such that the validity of the output is signaled using the Rust type system with potentially insufficient output space.
  • Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER.
  • Converts potentially-invalid UTF-16 to valid UTF-8 with errors replaced with the REPLACEMENT CHARACTER with potentially insufficient output space.
  • Copies ASCII from source to destination up to the first non-ASCII byte (or the end of the input if it is ASCII in its entirety).
  • Copies ASCII from source to destination zero-extending it to UTF-16 up to the first non-ASCII byte (or the end of the input if it is ASCII in its entirety).
  • Copies Basic Latin from source to destination narrowing it to ASCII up to the first non-Basic Latin code unit (or the end of the input if it is Basic Latin in its entirety).
  • Converts bytes whose unsigned value is interpreted as Unicode code point (i.e. U+0000 to U+00FF, inclusive) to UTF-8.
  • If the input is valid UTF-8 representing only Unicode code points from U+0000 to U+00FF, inclusive, converts the input into output that represents the value of each code point as the unsigned byte value of each output byte.
  • Replaces unpaired surrogates in the input with the REPLACEMENT CHARACTER.
  • Checks whether the buffer is all-ASCII.
  • Checks whether the buffer is all-Basic Latin (i.e. UTF-16 representing only ASCII characters).
  • Checks whether a scalar value triggers right-to-left processing.
  • Checks whether a valid UTF-8 buffer contains code points that trigger right-to-left processing.
  • Checks whether the buffer represents only code points less than or equal to U+00FF.
  • Checks whether a potentially-invalid UTF-8 buffer contains code points that trigger right-to-left processing.
  • Checks whether the buffer is valid UTF-8 representing only code points less than or equal to U+00FF.
  • Checks whether a UTF-16 buffer contains code points that trigger right-to-left processing.
  • Checks whether a UTF-16 code unit triggers right-to-left processing.
  • Checks whether the buffer represents only code point less than or equal to U+00FF.
  • Returns the index of first byte that starts a non-Latin1 byte sequence, or the length of the string if there are none.
  • Returns the index of first byte that starts an invalid byte sequence or a non-Latin1 byte sequence, or the length of the string if there are neither.
  • Returns the index of the first unpaired surrogate or, if the input is valid UTF-16 in its entirety, the length of the input.