simdutf8/lib.rs
1#![warn(unused_extern_crates)]
2#![warn(
3 clippy::all,
4 clippy::unwrap_used,
5 clippy::unnecessary_unwrap,
6 clippy::pedantic,
7 clippy::nursery
8)]
9#![allow(clippy::redundant_pub_crate)] // check is broken (see e.g. https://github.com/rust-lang/rust-clippy/issues/5369)
10#![allow(clippy::missing_const_for_fn)] // not necessary most of the times
11#![deny(missing_docs)]
12#![cfg_attr(not(feature = "std"), no_std)]
13#![cfg_attr(docsrs, feature(doc_cfg))]
14#![cfg_attr(
15 all(target_arch = "aarch64", feature = "aarch64_neon_prefetch"),
16 feature(stdarch_aarch64_prefetch)
17)]
18#![cfg_attr(feature = "portable_public_imp", feature(portable_simd))]
19#![cfg_attr(
20 all(
21 target_arch = "arm",
22 target_feature = "v7",
23 target_endian = "little",
24 not(target_feature = "neon"),
25 feature = "armv7_neon",
26 feature = "std"
27 ),
28 feature(stdarch_arm_feature_detection)
29)]
30#![cfg_attr(
31 all(
32 target_arch = "arm",
33 target_feature = "v7",
34 target_endian = "little",
35 feature = "armv7_neon",
36 any(feature = "std", feature = "public_imp", target_feature = "neon")
37 ),
38 feature(stdarch_arm_neon_intrinsics)
39)]
40#![cfg_attr(
41 all(
42 target_arch = "arm",
43 target_feature = "v7",
44 target_endian = "little",
45 feature = "armv7_neon",
46 any(feature = "std", feature = "public_imp", target_feature = "neon")
47 ),
48 feature(arm_target_feature)
49)]
50
51//! Blazingly fast API-compatible UTF-8 validation for Rust using SIMD extensions, based on the implementation from
52//! [simdjson](https://github.com/simdjson/simdjson). Originally ported to Rust by the developers of [simd-json.rs](https://simd-json.rs), but now heavily improved.
53//!
54//! ## Quick start
55//! Add the dependency to your Cargo.toml file:
56//! ```toml
57//! [dependencies]
58//! simdutf8 = "0.1.5"
59//! ```
60//!
61//! Use [`basic::from_utf8()`] as a drop-in replacement for `std::str::from_utf8()`.
62//!
63//! ```rust
64//! use simdutf8::basic::from_utf8;
65//!
66//! println!("{}", from_utf8(b"I \xE2\x9D\xA4\xEF\xB8\x8F UTF-8!").unwrap());
67//! ```
68//!
69//! If you need detailed information on validation failures, use [`compat::from_utf8()`]
70//! instead.
71//!
72//! ```rust
73//! use simdutf8::compat::from_utf8;
74//!
75//! let err = from_utf8(b"I \xE2\x9D\xA4\xEF\xB8 UTF-8!").unwrap_err();
76//! assert_eq!(err.valid_up_to(), 5);
77//! assert_eq!(err.error_len(), Some(2));
78//! ```
79//!
80//! ## APIs
81//!
82//! ### Basic flavor
83//! Use the `basic` API flavor for maximum speed. It is fastest on valid UTF-8, but only checks
84//! for errors after processing the whole byte sequence and does not provide detailed information if the data
85//! is not valid UTF-8. [`basic::Utf8Error`] is a zero-sized error struct.
86//!
87//! ### Compat flavor
88//! The `compat` flavor is fully API-compatible with `std::str::from_utf8()`. In particular, [`compat::from_utf8()`]
89//! returns a [`compat::Utf8Error`], which has [`valid_up_to()`](compat::Utf8Error#method.valid_up_to) and
90//! [`error_len()`](compat::Utf8Error#method.error_len) methods. The first is useful for verification of streamed data. The
91//! second is useful e.g. for replacing invalid byte sequences with a replacement character.
92//!
93//! It also fails early: errors are checked on the fly as the string is processed and once
94//! an invalid UTF-8 sequence is encountered, it returns without processing the rest of the data.
95//! This comes at a slight performance penalty compared to the [`basic`] API even if the input is valid UTF-8.
96//!
97//! ## Implementation selection
98//!
99//! ### X86
100//! The fastest implementation is usually selected at runtime using the `std::is_x86_feature_detected!` macro. The AVX 512
101//! implementation requires Rust 1.89 or late and is only selected if the CPU support the VBMI2 features to avoid throttling
102//! happening with CPUs before Intels Ice Lake microarchitecture.
103//!
104//! If you compile with `RUSTFLAGS="-C target-cpu=native"` on a recent x86-64 machine whichs support AVX 512 with Rust 1.89 or later,
105//! the AVX 512 implementation is selected at compile-time and runtime selection is disabled.
106//!
107//! For no-std support (compiled with `--no-default-features`) the implementation is always selected at compile time based on
108//! the targeted CPU. Use `RUSTFLAGS="-C target-feature=+avx2"` for the AVX 2 implementation or `RUSTFLAGS="-C target-feature=+sse4.2"`
109//! for the SSE 4.2 implementation. For AVX 512 use `RUSTFLAGS="-C target-feature=+avx512f,+avx512bw,+avx512vbmi,+avx512vbmi2"` with
110//! Rust 1.89 or later.
111//!
112//! ### ARM64
113//! The SIMD implementation is used automatically since Rust 1.61.
114//!
115//! ### WASM32
116//! For wasm32 support, the implementation is selected at compile time based on the presence of the `simd128` target feature.
117//! Use `RUSTFLAGS="-C target-feature=+simd128"` to enable the WASM SIMD implementation. WASM, at
118//! the time of this writing, doesn't have a way to detect SIMD through WASM itself. Although this capability
119//! is available in various WASM host environments (e.g., [wasm-feature-detect] in the web browser), there is no portable
120//! way from within the library to detect this.
121//!
122//! [wasm-feature-detect]: https://github.com/GoogleChromeLabs/wasm-feature-detect
123//!
124//! ### Access to low-level functionality
125//! If you want to be able to call a SIMD implementation directly, use the `public_imp` feature flag. The validation
126//! implementations are then accessible via [`basic::imp`] and [`compat::imp`]. Traits facilitating streaming validation are available
127//! there as well.
128//!
129//! ## Optimisation flags
130//! Do not use [`opt-level = "z"`](https://doc.rust-lang.org/cargo/reference/profiles.html), which prevents inlining and makes
131//! the code quite slow.
132//!
133//! ## Minimum Supported Rust Version (MSRV)
134//! This crate's minimum supported Rust version is 1.38.0.
135//!
136//! ## Algorithm
137//!
138//! See Validating UTF-8 In Less Than One Instruction Per Byte, Software: Practice and Experience 51 (5), 2021
139//! <https://arxiv.org/abs/2010.03090>
140
141pub mod basic;
142pub mod compat;
143mod implementation;