Crate simdutf

Unicode validation and transcoding at billions of characters per second.

This crate is the Rust binding of simdutf.

Compilation

This crate works out of the box as long as you have a C++11-compatible toolchain installed correctly.

simdutf links C++ standard library, which adds a dynamic linking dependency.

For more details, see simdutf documentation and cc documentation.

Here is an example for local benchmark:

export RUSTFLAGS='-C target-cpu=native'
export CXXFLAGS='-march=native'
cargo build --release

Structs

• Encoding
  The encoding of a string, defined as a bitflags type.
• Result
  The result type of validation and transcoding.

Enums

• ErrorCode
  The error code type of validation and transcoding.

Functions

• autodetect_encodings
  Autodetect the possible encodings of the input in one pass.
• autodetect_single_encoding
  Autodetect the encoding of the input.
• change_endianness_utf16^⚠
  Change the endianness of UTF-16 string.
• convert_utf8_to_utf16^⚠
  Convert possibly broken UTF-8 string into UTF-16 string.
• convert_utf8_to_utf16_with_errors^⚠
  Convert possibly broken UTF-8 string into UTF-16 string.
• convert_utf8_to_utf16be^⚠
  Convert possibly broken UTF-8 string into UTF-16BE string.
• convert_utf8_to_utf16be_with_errors^⚠
  Convert possibly broken UTF-8 string into UTF-16BE string.
• convert_utf8_to_utf16le^⚠
  Convert possibly broken UTF-8 string into UTF-16LE string.
• convert_utf8_to_utf16le_with_errors^⚠
  Convert possibly broken UTF-8 string into UTF-16LE string.
• convert_utf8_to_utf32^⚠
  Convert possibly broken UTF-8 string into UTF-32 string.
• convert_utf8_to_utf32_with_errors^⚠
  Convert possibly broken UTF-8 string into UTF-32 string.
• convert_utf16_to_utf8^⚠
  Convert possibly broken UTF-16 string into UTF-8 string.
• convert_utf16_to_utf8_with_errors^⚠
  Convert possibly broken UTF-16 string into UTF-8 string.
• convert_utf16_to_utf32^⚠
  Convert possibly broken UTF-16 string into UTF-32 string.
• convert_utf16_to_utf32_with_errors^⚠
  Convert possibly broken UTF-16 string into UTF-32 string.
• convert_utf16be_to_utf8^⚠
  Convert possibly broken UTF-16BE string into UTF-8 string.
• convert_utf16be_to_utf8_with_errors^⚠
  Convert possibly broken UTF-16BE string into UTF-8 string.
• convert_utf16be_to_utf32^⚠
  Convert possibly broken UTF-16BE string into UTF-32 string.
• convert_utf16be_to_utf32_with_errors^⚠
  Convert possibly broken UTF-16BE string into UTF-32 string.
• convert_utf16le_to_utf8^⚠
  Convert possibly broken UTF-16LE string into UTF-8 string.
• convert_utf16le_to_utf8_with_errors^⚠
  Convert possibly broken UTF-16LE string into UTF-8 string.
• convert_utf16le_to_utf32^⚠
  Convert possibly broken UTF-16LE string into UTF-32 string.
• convert_utf16le_to_utf32_with_errors^⚠
  Convert possibly broken UTF-16LE string into UTF-32 string.
• convert_utf32_to_utf8^⚠
  Convert possibly broken UTF-32 string into UTF-8 string.
• convert_utf32_to_utf8_with_errors^⚠
  Convert possibly broken UTF-32 string into UTF-8 string.
• convert_utf32_to_utf16^⚠
  Convert possibly broken UTF-32 string into UTF-16 string.
• convert_utf32_to_utf16_with_errors^⚠
  Convert possibly broken UTF-32 string into UTF-16 string.
• convert_utf32_to_utf16be^⚠
  Convert possibly broken UTF-32 string into UTF-16BE string.
• convert_utf32_to_utf16be_with_errors^⚠
  Convert possibly broken UTF-32 string into UTF-16BE string.
• convert_utf32_to_utf16le^⚠
  Convert possibly broken UTF-32 string into UTF-16LE string.
• convert_utf32_to_utf16le_with_errors^⚠
  Convert possibly broken UTF-32 string into UTF-16LE string.
• convert_valid_utf8_to_utf16^⚠
  Convert valid UTF-8 string into UTF-16 string.
• convert_valid_utf8_to_utf16be^⚠
  Convert valid UTF-8 string into UTF-16BE string.
• convert_valid_utf8_to_utf16le^⚠
  Convert valid UTF-8 string into UTF-16LE string.
• convert_valid_utf8_to_utf32^⚠
  Convert valid UTF-8 string into UTF-32 string.
• convert_valid_utf16_to_utf8^⚠
  Convert valid UTF-16 string into UTF-8 string.
• convert_valid_utf16_to_utf32^⚠
  Convert valid UTF-16 string into UTF-32 string.
• convert_valid_utf16be_to_utf8^⚠
  Convert valid UTF-16BE string into UTF-8 string.
• convert_valid_utf16be_to_utf32^⚠
  Convert valid UTF-16BE string into UTF-32 string.
• convert_valid_utf16le_to_utf8^⚠
  Convert valid UTF-16LE string into UTF-8 string.
• convert_valid_utf16le_to_utf32^⚠
  Convert valid UTF-16LE string into UTF-32 string.
• convert_valid_utf32_to_utf8^⚠
  Convert valid UTF-32 string into UTF-8 string.
• convert_valid_utf32_to_utf16^⚠
  Convert valid UTF-32 string into UTF-16 string.
• convert_valid_utf32_to_utf16be^⚠
  Convert valid UTF-32 string into UTF-16BE string.
• convert_valid_utf32_to_utf16le^⚠
  Convert valid UTF-32 string into UTF-16LE string.
• count_utf8^⚠
  Count the number of code points in the UTF-8 string.
• count_utf16^⚠
  Count the number of code points in the UTF-16 string.
• count_utf16be^⚠
  Count the number of code points in the UTF-16BE string.
• count_utf16le^⚠
  Count the number of code points in the UTF-16LE string.
• utf8_length_from_utf16^⚠
  Count the number of code units that the UTF-16 string would require in UTF-8 format.
• utf8_length_from_utf16be^⚠
  Count the number of code units that the UTF-16BE string would require in UTF-8 format.
• utf8_length_from_utf16le^⚠
  Count the number of code units that the UTF-16LE string would require in UTF-8 format.
• utf8_length_from_utf32^⚠
  Count the number of code units that the UTF-32 string would require in UTF-8 format.
• utf16_length_from_utf8^⚠
  Count the number of code units that the UTF-8 string would require in UTF-16 format.
• utf16_length_from_utf32^⚠
  Count the number of code units that the UTF-32 string would require in UTF-16 format.
• utf32_length_from_utf8^⚠
  Count the number of code units that the UTF-8 string would require in UTF-32 format.
• utf32_length_from_utf16^⚠
  Count the number of code units that the UTF-16 string would require in UTF-32 format.
• utf32_length_from_utf16be^⚠
  Count the number of code units that the UTF-16BE string would require in UTF-32 format.
• utf32_length_from_utf16le^⚠
  Count the number of code units that the UTF-16LE string would require in UTF-32 format.
• validate_ascii
  Validate the ASCII string.
• validate_ascii_with_errors
  Validate the ASCII string.
• validate_utf8
  Validate the UTF-8 string.
• validate_utf8_with_errors
  Validate the UTF-8 string.
• validate_utf16
  Validate the UTF-16 string.
• validate_utf16_with_errors
  Validate the UTF-16 string.
• validate_utf16be
  Validate the UTF-16BE string.
• validate_utf16be_with_errors
  Validate the UTF-16BE string.
• validate_utf16le
  Validate the UTF-16LE string.
• validate_utf16le_with_errors
  Validate the UTF-16LE string.
• validate_utf32
  Validate the UTF-32 string.
• validate_utf32_with_errors
  Validate the UTF-32 string.