fearless_simd/lib.rs
1// Copyright 2024 the Fearless_SIMD Authors
2// SPDX-License-Identifier: Apache-2.0 OR MIT
3
4//! A helper library to make SIMD more friendly.
5//!
6//! Fearless SIMD exposes safe SIMD with ergonomic multi-versioning in Rust.
7//!
8//! Fearless SIMD uses "marker values" which serve as proofs of which target features are available on the current CPU.
9//! These each implement the [`Simd`] trait, which exposes a core set of SIMD operations which are implemented as
10//! efficiently as possible on each target platform.
11//!
12//! Additionally, there are types for packed vectors of a specific width and element type (such as [`f32x4`]).
13//! Fearless SIMD does not currently support vectors of less than 128 bits.
14//! These vector types implement some standard arithmetic traits (i.e. they can be added together using
15//! `+`, multiplied by a scalar using `*`, among others), which are implemented as efficiently
16//! as possible using SIMD instructions.
17//! These can be created in a SIMD context using the [`SimdFrom`] trait, or the
18//! [`from_slice`][SimdBase::from_slice] associated function.
19//!
20//! To call a function with the best available target features and get the associated `Simd`
21//! implementation, use the [`dispatch!()`] macro:
22//!
23//! ```rust
24//! use fearless_simd::{Level, Simd, dispatch};
25//!
26//! #[inline(always)]
27//! fn sigmoid<S: Simd>(simd: S, x: &[f32], out: &mut [f32]) { /* ... */ }
28//!
29//! // The stored level, which you should only construct once in your application.
30//! let level = Level::new();
31//!
32//! dispatch!(level, simd => sigmoid(simd, &[/*...*/], &mut [/*...*/]));
33//! ```
34//!
35//! A few things to note:
36//!
37//! 1) `sigmoid` is generic over any `Simd` type.
38//! 2) The [`dispatch`] macro is used to invoke the given function with the target features associated with the supplied [`Level`].
39//! 3) The function or closure passed to [`dispatch!()`] should be `#[inline(always)]`.
40//! The performance of the SIMD implementation may be poor if that isn't the case. See [the section on inlining for details](#inlining)
41//!
42//! The first parameter to [`dispatch!()`] is the [`Level`].
43//! If you are writing an application, you should create this once (using [`Level::new`]), and pass it to any function which wants to use SIMD.
44//! This type stores which instruction sets are available for the current process, which is used
45//! in the macro to dispatch to the most optimal variant of the supplied function for this process.
46//!
47//! # Inlining
48//!
49//! Fearless SIMD relies heavily on Rust's inlining support to create functions which have the
50//! given target features enabled.
51//! As such, most functions which you write when using Fearless SIMD should have the `#[inline(always)]` attribute.
52//!
53//! There is a rule of thumb for how to achieve things in Fearless SIMD:
54//!
55//! - All SIMD functions need `#[inline(always)]`.
56//! - Use [`dispatch!`] when calling SIMD code from non-SIMD code.
57//! - Use [`vectorize()`](Simd::vectorize) when calling SIMD from SIMD if you don't want to force inlining.
58//!
59//! We currently don't have docs explaining why this is the case.
60//! You can read [this Zulip conversation](https://xi.zulipchat.com/#narrow/channel/514230-simd/topic/inlining/with/546913433)
61//! for some train of thought explanation.
62//!
63//! <!--
64//! TODO: Also have concrete examples of each of these.
65//!
66//! TODO: This is a really subtle point, and we do need there to be a well-written explanation available.
67//! E.g. We might want names for these, e.g.:
68//!
69//! # Kernels vs not kernels
70//!
71//! TODO: Talk about writing versions of functions which can be called in other `S: Simd` functions.
72//! -->
73//!
74//! # WebAssembly
75//!
76//! WASM SIMD doesn't have feature detection, and so you need to compile two versions of your bundle for WASM, one with SIMD and one without,
77//! then select the appropriate one for your user's browser. This can be done via [the `wasm-feature-detect`
78//! library](https://github.com/GoogleChromeLabs/wasm-feature-detect).
79//!
80//! You can compile WebAssembly with the SIMD128 feature enabled via the `RUSTFLAGS` environment variable
81//! (`RUSTFLAGS="-Ctarget-feature=+simd128"`), or by adding the compiler flags in your [Cargo
82//! config.toml](https://doc.rust-lang.org/cargo/reference/config.html):
83//!
84//! ```toml
85//! [target.'cfg(target_arch = "wasm32")']
86//! rustflags = ["-Ctarget-feature=+simd128"]
87//! rustdocflags = ["-Ctarget-feature=+simd128"]
88//! ```
89//!
90//! If you want to compile both SIMD and non-SIMD versions of your WebAssembly library, your best option right now is to create a shell script
91//! that builds it once with the `RUSTFLAGS` specified, and once without. [Cargo currently does not allow specifying compiler flags
92//! per-profile.](https://github.com/rust-lang/cargo/issues/10271)
93//!
94//! ## Relaxed SIMD
95//!
96//! Fearless SIMD can make use of the [relaxed SIMD](https://github.com/WebAssembly/relaxed-simd/blob/main/proposals/relaxed-simd/Overview.md)
97//! WebAssembly instructions, if the requisite target feature is enabled. These instructions can return implementation-dependent results
98//! depending on what is fastest on the underlying hardware. They are only used for operations where we already give hardware-dependent results.
99//!
100//! At the time of writing, relaxed SIMD is only supported in Chrome. To make use of it, you'll need to build two versions of your library, one
101//! with relaxed SIMD enabled (`RUSTFLAGS="-Ctarget-feature=+simd128,+relaxed-simd"`) and one with it disabled, and then feature-detect at
102//! runtime.
103//!
104//! # Credits
105//!
106//! This crate was inspired by [`pulp`], [`std::simd`], among others in the Rust ecosystem, though makes many decisions differently.
107//! It benefited from conversations with Luca Versari, though he is not responsible for any of the mistakes or bad decisions.
108//!
109//! # Feature Flags
110//!
111//! The following crate [feature flags](https://doc.rust-lang.org/cargo/reference/features.html#dependency-features) are available:
112//!
113//! - `std` (enabled by default): Get floating point functions from the standard library (likely using your target's libc).
114//! Also allows using [`Level::new`] on all platforms, to detect which target features are enabled.
115//! - `libm`: Use floating point implementations from [libm].
116//! - `safe_wrappers`: Include safe wrappers for (some) target feature specific intrinsics,
117//! beyond the basic SIMD operations abstracted on all platforms.
118//! - `force_support_fallback`: Force scalar fallback, to be supported, even if your compilation target has a better baseline.
119//!
120//! At least one of `std` and `libm` is required; `std` overrides `libm`.
121//!
122//! [`pulp`]: https://crates.io/crates/pulp
123// LINEBENDER LINT SET - lib.rs - v3
124// See https://linebender.org/wiki/canonical-lints/
125// These lints shouldn't apply to examples or tests.
126#![cfg_attr(not(test), warn(unused_crate_dependencies))]
127// These lints shouldn't apply to examples.
128#![warn(clippy::print_stdout, clippy::print_stderr)]
129// Targeting e.g. 32-bit means structs containing usize can give false positives for 64-bit.
130#![cfg_attr(target_pointer_width = "64", warn(clippy::trivially_copy_pass_by_ref))]
131// END LINEBENDER LINT SET
132#![cfg_attr(docsrs, feature(doc_cfg))]
133#![allow(non_camel_case_types, reason = "TODO")]
134#![expect(clippy::unused_unit, reason = "easier for code generation")]
135#![no_std]
136
137#[cfg(feature = "std")]
138extern crate std;
139
140#[cfg(all(not(feature = "libm"), not(feature = "std")))]
141compile_error!("fearless_simd requires either the `std` or `libm` feature");
142
143// Suppress the unused_crate_dependencies lint when both std and libm are specified.
144#[cfg(all(feature = "std", feature = "libm"))]
145use libm as _;
146
147pub mod core_arch;
148mod impl_macros;
149
150mod generated;
151mod macros;
152mod support;
153mod traits;
154
155pub use generated::*;
156pub use traits::*;
157
158/// This prelude module re-exports every SIMD trait defined in this library. It's useful for accessing trait methods.
159///
160/// Only traits are exported through the prelude; types must be exported separately.
161pub mod prelude {
162 pub use crate::generated::simd_trait::*;
163 pub use crate::traits::*;
164}
165
166/// Implementations of [`Simd`] for 64 bit ARM.
167#[cfg(target_arch = "aarch64")]
168pub mod aarch64 {
169 pub use crate::generated::Neon;
170}
171
172/// Implementations of [`Simd`] for webassembly.
173#[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]
174pub mod wasm32 {
175 pub use crate::generated::WasmSimd128;
176}
177
178/// Implementations of [`Simd`] on x86 architectures (both 32 and 64 bit).
179#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
180pub mod x86 {
181 pub use crate::generated::Avx2;
182 pub use crate::generated::Sse4_2;
183}
184
185/// The level enum with the specific SIMD capabilities available.
186///
187/// The contained values serve as a proof that the associated target
188/// feature is available.
189#[derive(Clone, Copy, Debug)]
190#[non_exhaustive]
191pub enum Level {
192 /// Scalar fallback level, i.e. no supported SIMD features are to be used.
193 ///
194 /// This can be created with [`Level::fallback`].
195 // We only want to compile the fallback implementation if:
196 // - We're on a supported architecture, but don't statically support the lowest alternative level; OR
197 // - We're on an unsupported architecture; OR
198 // - The fallback is forcibly enabled
199 #[cfg(any(
200 all(target_arch = "aarch64", not(target_feature = "neon")),
201 all(
202 any(target_arch = "x86", target_arch = "x86_64"),
203 not(all(
204 target_feature = "sse4.2",
205 target_feature = "cmpxchg16b",
206 target_feature = "popcnt"
207 ))
208 ),
209 all(target_arch = "wasm32", not(target_feature = "simd128")),
210 not(any(
211 target_arch = "x86",
212 target_arch = "x86_64",
213 target_arch = "aarch64",
214 target_arch = "wasm32"
215 )),
216 feature = "force_support_fallback"
217 ))]
218 Fallback(Fallback),
219 /// The Neon instruction set on 64 bit ARM.
220 #[cfg(target_arch = "aarch64")]
221 Neon(Neon),
222 /// The SIMD 128 instructions on 32-bit WebAssembly.
223 #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]
224 WasmSimd128(WasmSimd128),
225 /// The SSE4.2 instruction set on (32 and 64 bit) x86, plus `popcnt` and `cmpxchg16b`.
226 /// Also known as x86-64-v2.
227 ///
228 /// All production CPUs with SSE4.2 also support the other two extensions, so it is safe to require them.
229 // We don't need to support this if the compilation target definitely supports something better.
230 #[cfg(all(
231 any(target_arch = "x86", target_arch = "x86_64"),
232 not(all(
233 target_feature = "avx2",
234 target_feature = "bmi1",
235 target_feature = "bmi2",
236 target_feature = "cmpxchg16b",
237 target_feature = "f16c",
238 target_feature = "fma",
239 target_feature = "lzcnt",
240 target_feature = "movbe",
241 target_feature = "popcnt",
242 target_feature = "xsave"
243 ))
244 ))]
245 Sse4_2(Sse4_2),
246 /// The AVX2 and FMA instruction set on (32 and 64 bit) x86, plus the other instructions
247 /// guaranteed to be available on AVX2+FMA CPUs. Also known as x86-64-v3.
248 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
249 Avx2(Avx2),
250 // If new variants are added, make sure to handle them in `Level::dispatch`
251 // and `dispatch!()`
252}
253
254impl Level {
255 /// Detect the available features on the current CPU, and returns the best level.
256 ///
257 /// If no SIMD instruction set is available, a scalar fallback will be used instead.
258 ///
259 /// This function requires the standard library, to use the
260 /// [`is_x86_feature_detected`](std::arch::is_x86_feature_detected)
261 /// or [`is_aarch64_feature_detected`](std::arch::is_aarch64_feature_detected).
262 /// On wasm32, this requirement does not apply, so the standard library isn't required.
263 ///
264 /// Note that in most cases, this function should only be called by end-user applications.
265 /// Libraries should instead accept a `Level` argument, probably as they are
266 /// creating their data structures, then storing the level for any computations.
267 /// Libraries which wish to abstract away SIMD usage for their common-case clients,
268 /// should make their non-`Level` entrypoint match this function's `cfg`; to instead
269 /// handle this at runtime, they can use [`try_detect`](Self::try_detect),
270 /// handling the `None` case as they deem fit (probably panicking).
271 /// This strategy avoids users of the library inadvertently using the fallback level,
272 /// even if the requisite target features are available.
273 ///
274 /// If you are on an embedded device where these macros are not supported,
275 /// you should construct the relevant variants yourself, using whatever
276 /// way your specific chip supports accessing the current level.
277 ///
278 /// This value should be passed to [`dispatch!()`].
279 #[cfg(any(feature = "std", target_arch = "wasm32"))]
280 #[must_use]
281 #[expect(
282 clippy::new_without_default,
283 reason = "The `Level::new()` function is not always available, and we also want to be explicit about when runtime feature detection happens"
284 )]
285 pub fn new() -> Self {
286 #[cfg(target_arch = "aarch64")]
287 if std::arch::is_aarch64_feature_detected!("neon") {
288 return unsafe { Self::Neon(Neon::new_unchecked()) };
289 }
290 #[cfg(target_arch = "wasm32")]
291 {
292 // WASM always either has the SIMD feature compiled in or not.
293 #[cfg(target_feature = "simd128")]
294 return Self::WasmSimd128(WasmSimd128::new_unchecked());
295 }
296 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
297 {
298 // Feature list sourced from `rustc --print=cfg --target x86_64-unknown-linux-gnu -C target-cpu=x86-64-v3`
299 // However, the following features are implied by avx2 and do not need to be spelled out:
300 // avx,fxsr,sse,sse2,sse3,sse4.1,sse4.2,ssse3
301 // This can be verified by running:
302 // rustc --print=cfg --target x86_64-unknown-linux-gnu -C target-feature='+avx2'
303 if std::arch::is_x86_feature_detected!("avx2")
304 && std::arch::is_x86_feature_detected!("bmi1")
305 && std::arch::is_x86_feature_detected!("bmi2")
306 && std::arch::is_x86_feature_detected!("cmpxchg16b")
307 && std::arch::is_x86_feature_detected!("f16c")
308 && std::arch::is_x86_feature_detected!("fma")
309 && std::arch::is_x86_feature_detected!("lzcnt")
310 && std::arch::is_x86_feature_detected!("movbe")
311 && std::arch::is_x86_feature_detected!("popcnt")
312 && std::arch::is_x86_feature_detected!("xsave")
313 {
314 return unsafe { Self::Avx2(Avx2::new_unchecked()) };
315 // All x86 CPUs that ever shipped with sse4.2 also have cmpxchg16b and popcnt:
316 // Intel Nehalem, AMD Bulldozer and VIA Isaiah II were the first with SSE4.2
317 // and have these extensions already.
318 } else if std::arch::is_x86_feature_detected!("sse4.2")
319 && std::arch::is_x86_feature_detected!("cmpxchg16b")
320 && std::arch::is_x86_feature_detected!("popcnt")
321 {
322 #[cfg(not(all(
323 target_feature = "avx2",
324 target_feature = "bmi1",
325 target_feature = "bmi2",
326 target_feature = "cmpxchg16b",
327 target_feature = "f16c",
328 target_feature = "fma",
329 target_feature = "lzcnt",
330 target_feature = "movbe",
331 target_feature = "popcnt",
332 target_feature = "xsave"
333 )))]
334 return unsafe { Self::Sse4_2(Sse4_2::new_unchecked()) };
335 }
336 }
337 #[cfg(any(
338 all(target_arch = "aarch64", not(target_feature = "neon")),
339 all(
340 any(target_arch = "x86", target_arch = "x86_64"),
341 not(all(
342 target_feature = "sse4.2",
343 target_feature = "cmpxchg16b",
344 target_feature = "popcnt"
345 ))
346 ),
347 all(target_arch = "wasm32", not(target_feature = "simd128")),
348 not(any(
349 target_arch = "x86",
350 target_arch = "x86_64",
351 target_arch = "aarch64",
352 target_arch = "wasm32"
353 )),
354 ))]
355 {
356 return Self::Fallback(Fallback::new());
357 }
358 #[allow(
359 unreachable_code,
360 reason = "`is_x86_feature_detected` or equivalents will have returned `true`, or Fallback was used."
361 )]
362 {
363 unreachable!()
364 }
365 }
366
367 /// Get the target feature level suitable for this run.
368 ///
369 /// Should be used in libraries if they wish to handle the case where
370 /// target features cannot be detected at runtime.
371 /// Most users should prefer [`new`](Self::new).
372 /// This is discussed in more detail in `new`'s documentation.
373 #[allow(clippy::allow_attributes, reason = "Only needed in some cfgs.")]
374 #[allow(unreachable_code, reason = "Fallback unreachable in some cfgs.")]
375 pub fn try_detect() -> Option<Self> {
376 #[cfg(any(feature = "std", target_arch = "wasm32"))]
377 return Some(Self::new());
378 None
379 }
380
381 /// Check whether this is the `Fallback` level; that is, whether no better feature level could
382 /// be statically or dynamically detected. This is useful if there's a scalarized version of
383 /// your algorithm that runs faster if SIMD isn't supported.
384 ///
385 /// This method is always available, even in cases where `Fallback` is not; for instance, if
386 /// you're targeting a platform that always supports some level of SIMD. In such cases, it will
387 /// always return false.
388 pub fn is_fallback(self) -> bool {
389 #[cfg(any(
390 all(target_arch = "aarch64", not(target_feature = "neon")),
391 all(
392 any(target_arch = "x86", target_arch = "x86_64"),
393 not(all(
394 target_feature = "sse4.2",
395 target_feature = "cmpxchg16b",
396 target_feature = "popcnt"
397 ))
398 ),
399 all(target_arch = "wasm32", not(target_feature = "simd128")),
400 not(any(
401 target_arch = "x86",
402 target_arch = "x86_64",
403 target_arch = "aarch64",
404 target_arch = "wasm32"
405 )),
406 feature = "force_support_fallback"
407 ))]
408 return matches!(self, Self::Fallback(_));
409
410 #[allow(unreachable_code, reason = "Fallback unreachable in some cfgs.")]
411 false
412 }
413
414 /// If this is a proof that Neon (or better) is available, access that instruction set.
415 ///
416 /// This method should be preferred over matching against the `Neon` variant of self,
417 /// because if Fearless SIMD gets support for an instruction set which is a superset of Neon,
418 /// this method will return a value even if that "better" instruction set is available.
419 ///
420 /// This can be used in combination with the `safe_wrappers` feature to gain checked access to
421 /// the level-specific SIMD capabilities.
422 #[cfg(target_arch = "aarch64")]
423 #[inline]
424 pub fn as_neon(self) -> Option<Neon> {
425 #[allow(
426 unreachable_patterns,
427 reason = "On machines which statically support `neon`, there is only one variant."
428 )]
429 match self {
430 Self::Neon(neon) => Some(neon),
431 _ => None,
432 }
433 }
434
435 /// If this is a proof that SIMD 128 (or better) is available, access that instruction set.
436 ///
437 /// This method should be preferred over matching against the `WasmSimd128` variant of self,
438 /// because if Fearless SIMD gets support for an instruction set which is a superset of SIMD 128,
439 /// this method will return a value even if that "better" instruction set is available.
440 ///
441 /// This can be used in combination with the `safe_wrappers` feature to gain checked access to
442 /// the level-specific SIMD capabilities.
443 #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]
444 #[inline]
445 pub fn as_wasm_simd128(self) -> Option<WasmSimd128> {
446 #[allow(
447 unreachable_patterns,
448 reason = "On machines which statically support `simd128`, there is only one variant."
449 )]
450 match self {
451 Self::WasmSimd128(simd128) => Some(simd128),
452 _ => None,
453 }
454 }
455
456 /// If this is a proof that SSE4.2 (or better) is available, access that instruction set.
457 ///
458 /// This method should be preferred over matching against the `Sse4_2` variant of self,
459 /// because if Fearless SIMD gets support for an instruction set which is a superset of SSE4.2,
460 /// this method will return a value even if that "better" instruction set is available.
461 ///
462 /// This can be used in combination with the `safe_wrappers` feature to gain checked access to
463 /// the level-specific SIMD capabilities.
464 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
465 #[inline]
466 pub fn as_sse4_2(self) -> Option<Sse4_2> {
467 match self {
468 // Safety: The Avx2 struct represents the `avx2` and `fma` target features being enabled.
469 // The `avx2` target feature *also* implicitly enables the "sse4.2" target feature, which is
470 // the only target feature required to make our Sse4_2 token.
471 Self::Avx2(_avx) => unsafe { Some(Sse4_2::new_unchecked()) },
472 #[cfg(not(all(
473 target_feature = "avx2",
474 target_feature = "bmi1",
475 target_feature = "bmi2",
476 target_feature = "cmpxchg16b",
477 target_feature = "f16c",
478 target_feature = "fma",
479 target_feature = "lzcnt",
480 target_feature = "movbe",
481 target_feature = "popcnt",
482 target_feature = "xsave"
483 )))]
484 Self::Sse4_2(sse42) => Some(sse42),
485 #[allow(
486 unreachable_patterns,
487 reason = "This arm is reachable on baseline x86/x86_64."
488 )]
489 _ => None,
490 }
491 }
492
493 /// If this is a proof that AVX2 and FMA (or better) is available, access that instruction set.
494 ///
495 /// This method should be preferred over matching against the `AVX2` variant of self,
496 /// because if Fearless SIMD gets support for an instruction set which is a superset of AVX2,
497 /// this method will return a value even if that "better" instruction set is available.
498 ///
499 /// This can be used in combination with the `safe_wrappers` feature to gain checked access to
500 /// the level-specific SIMD capabilities.
501 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
502 #[inline]
503 pub fn as_avx2(self) -> Option<Avx2> {
504 #[allow(
505 unreachable_patterns,
506 reason = "On machines which statically support `avx2`, there is only one variant."
507 )]
508 match self {
509 Self::Avx2(avx2) => Some(avx2),
510 _ => None,
511 }
512 }
513
514 /// Get the strongest statically supported SIMD level.
515 ///
516 /// That is, if your compilation run ambiently declares that a target feature is enabled,
517 /// this method will take that into account.
518 /// In most cases, you should use [`Level::new`] or [`Level::try_detect`].
519 /// This method is mainly useful for libraries, where:
520 ///
521 /// 1) Your crate features request that you not use the standard library, i.e. doesn't enable
522 /// your `"std"` crate feature reason (so you can't use [`Level::new`] and
523 /// [`Level::try_detect`] returns `None`); AND
524 /// 2) Your caller does not provide a [`Level`]; AND
525 /// 3) The library doesn't want to panic when it can't find a SIMD level.
526 ///
527 /// Note that in these cases, the library should clearly inform the integrator
528 /// that it is using a fallback and so not getting optimal performance (e.g. by panicking if
529 /// `debug_assertions` are enabled, and emitting a log with the "error" level otherwise).
530 /// The messages given should also provide actionable fixes, such as pointing to the
531 /// entry-point which provides a `Level`, or your `"std"` feature.
532 ///
533 /// Note that this is unaffected by the `force-support-fallback` feature.
534 /// Instead, you should use [`Level::fallback`] if you require the fallback level.
535 pub const fn baseline() -> Self {
536 // TODO: How do we possibly test that this method works in all cases?
537 // Note that you can use the `check_targets.sh` script to at least ensure that it compiles in all reasonable cases.
538 #[cfg(not(any(
539 target_arch = "x86",
540 target_arch = "x86_64",
541 target_arch = "aarch64",
542 target_arch = "wasm32"
543 )))]
544 {
545 return Self::Fallback(Fallback::new());
546 }
547 #[cfg(target_arch = "aarch64")]
548 {
549 #[cfg(target_feature = "neon")]
550 return unsafe { Self::Neon(Neon::new_unchecked()) };
551 #[cfg(not(target_feature = "neon"))]
552 return Self::Fallback(Fallback::new());
553 }
554 #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
555 {
556 #[cfg(all(
557 target_feature = "avx2",
558 target_feature = "bmi1",
559 target_feature = "bmi2",
560 target_feature = "cmpxchg16b",
561 target_feature = "f16c",
562 target_feature = "fma",
563 target_feature = "lzcnt",
564 target_feature = "movbe",
565 target_feature = "popcnt",
566 target_feature = "xsave"
567 ))]
568 return unsafe { Self::Avx2(Avx2::new_unchecked()) };
569 #[cfg(all(
570 all(
571 target_feature = "sse4.2",
572 target_feature = "cmpxchg16b",
573 target_feature = "popcnt"
574 ),
575 not(all(
576 target_feature = "avx2",
577 target_feature = "bmi1",
578 target_feature = "bmi2",
579 target_feature = "cmpxchg16b",
580 target_feature = "f16c",
581 target_feature = "fma",
582 target_feature = "lzcnt",
583 target_feature = "movbe",
584 target_feature = "popcnt",
585 target_feature = "xsave"
586 ))
587 ))]
588 return unsafe { Self::Sse4_2(Sse4_2::new_unchecked()) };
589 #[cfg(not(all(
590 target_feature = "sse4.2",
591 target_feature = "cmpxchg16b",
592 target_feature = "popcnt"
593 )))]
594 return Self::Fallback(Fallback::new());
595 }
596 #[cfg(target_arch = "wasm32")]
597 {
598 #[cfg(target_feature = "simd128")]
599 return Self::WasmSimd128(WasmSimd128::new_unchecked());
600 #[cfg(not(target_feature = "simd128"))]
601 return Self::Fallback(Fallback::new());
602 }
603 }
604
605 /// Create a scalar fallback level, which uses no SIMD instructions.
606 ///
607 /// This is primarily intended for tests; most users should prefer [`Level::new`] or [`Level::baseline`].
608 ///
609 /// Note that enabling the scalar fallback does *not* mean that the fallback branch will not
610 /// contain SIMD instructions. This is because the "ambient" compilation environment has SIMD
611 /// instructions available, which may be utilised by LLVM to auto-vectorise that path.
612 #[inline]
613 #[cfg(feature = "force_support_fallback")]
614 pub const fn fallback() -> Self {
615 Self::Fallback(Fallback::new())
616 }
617
618 /// Dispatch `f` to a context where the target features which this `Level` proves are available are [enabled].
619 ///
620 /// Most users of Fearless SIMD should prefer to use [`dispatch!()`] to
621 /// explicitly vectorize a function. That has a better developer experience
622 /// than an implementation of `WithSimd`, and is less likely to miss a vectorization
623 /// opportunity.
624 ///
625 /// This has two use cases:
626 /// 1) To call a manually written implementation of [`WithSimd`].
627 /// 2) To ask the compiler to auto-vectorize scalar code.
628 ///
629 /// For the second case to work, the provided function *must* be attributed with `#[inline(always)]`.
630 /// Note also that any calls that function makes to other functions will likely not be auto-vectorized,
631 /// unless they are also `#[inline(always)]`.
632 ///
633 /// [enabled]: https://doc.rust-lang.org/reference/attributes/codegen.html#the-target_feature-attribute
634 #[inline]
635 #[expect(
636 unreachable_patterns,
637 reason = "Level is `non_exhaustive`, but we are in the crate it's defined."
638 )]
639 pub fn dispatch<W: WithSimd>(self, f: W) -> W::Output {
640 dispatch!(self, simd => f.with_simd(simd))
641 }
642}
643
644#[cfg(test)]
645mod tests {
646 use crate::Level;
647
648 const fn assert_is_send_sync<T: Send + Sync>() {}
649 /// If this test compiles, we know that [`Level`] is properly `Send` and `Sync`.
650 #[test]
651 fn level_is_send_sync() {
652 assert_is_send_sync::<Level>();
653 }
654}