atomic_maybe_uninit/

lib.rs

// SPDX-License-Identifier: Apache-2.0 OR MIT

/*!
<!-- Note: Document from sync-markdown-to-rustdoc:start through sync-markdown-to-rustdoc:end
     is synchronized from README.md. Any changes to that range are not preserved. -->
<!-- tidy:sync-markdown-to-rustdoc:start -->

Atomic operations on potentially uninitialized integers.

## Motivation

Copying types containing uninitialized bytes (e.g., padding), via the standard library's atomic types
is [undefined behavior because the copy goes through integers][undefined-behavior].

This crate provides a way to soundly perform such operations.

## Platform Support

Currently, all CPU architectures supported by Rust (x86, x86_64, Arm, AArch64, Arm64EC, RISC-V, LoongArch, s390x, PowerPC, MIPS, SPARC, AVR, MSP430, Hexagon, M68k, C-SKY, and Xtensa) are supported.
(You can use `cfg_{has,no}_*` macros to write code based on which primitive sizes are available for the current target and Rust version.)

| target_arch                                 | primitives                                          | load/store | swap/CAS |
| ------------------------------------------- | --------------------------------------------------- |:----------:|:--------:|
| x86                                         | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓        |
| x86_64                                      | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓        |
| x86_64 (+cmpxchg16b) \[2]                   | i128,u128                                           | ✓          | ✓        |
| arm (v6+ or Linux/Android)                  | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓\[1]    |
| arm (except for M-profile) \[3]             | i64,u64                                             | ✓          | ✓        |
| aarch64                                     | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64,i128,u128 | ✓          | ✓        |
| arm64ec \[10]                               | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64,i128,u128 | ✓          | ✓        |
| riscv32                                     | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓\[1]    |
| riscv32 (+zacas) \[4]                       | i64,u64                                             | ✓          | ✓        |
| riscv64                                     | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓\[1]    |
| riscv64 (+zacas) \[4]                       | i128,u128                                           | ✓          | ✓        |
| loongarch64                                 | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓        |
| loongarch64 (+scq) \[5]                     | i128,u128                                           | ✓          | ✓        |
| loongarch32 \[11]                           | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓        |
| s390x \[10]                                 | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64,i128,u128 | ✓          | ✓        |
| powerpc \[12]                               | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓        |
| powerpc64 \[12]                             | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓        |
| powerpc64 (+quadword-atomics) \[6] \[12]    | i128,u128                                           | ✓          | ✓        |
| mips / mips32r6 (experimental \[13])        | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓        |
| mips64 / mips64r6 (experimental \[13])      | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓        |
| sparc (experimental \[13])                  | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓\[1]    |
| sparc (+v8plus) \[8] (experimental \[13])   | i64,u64                                             | ✓          | ✓        |
| sparc64 (experimental \[13])                | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓        |
| avr (experimental \[13])                    | isize,usize,i8,u8,i16,u16                           | ✓          | ✓        |
| msp430 (experimental \[13])                 | isize,usize,i8,u8,i16,u16                           | ✓          | ✓        |
| hexagon (experimental \[13])                | isize,usize,i8,u8,i16,u16,i32,u32,i64,u64           | ✓          | ✓        |
| m68k (experimental \[13])                   | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓\[1]    |
| m68k (+isa-68020) \[9] (experimental \[13]) | i64,u64                                             | ✓          | ✓        |
| csky (experimental \[13])                   | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓\[1]    |
| xtensa (experimental \[13])                 | isize,usize,i8,u8,i16,u16,i32,u32                   | ✓          | ✓\[1]    |

\[1] Arm's RMW operations are not available on Armv6-M (thumbv6m). RISC-V's RMW operations are not available on targets without the A (or G which means IMAFD) or Zalrsc or Zacas extension, such as riscv32i, riscv32imc, etc. 32-bit SPARC's RMW operations requires `v9` or `leoncasa` target feature (enabled by default on Linux). M68k's atomic RMW operations requires target-cpu M68020+ (enabled by default on Linux). C-SKY's atomic RMW operations requires target-cpu ck860\* or c860\* (enabled by default on the hard-float target). Xtensa's atomic RMW operations are not available on esp32s2.<br>
\[2] Requires `cmpxchg16b` target feature (enabled by default on Apple, Windows (except Windows 7), and Fuchsia targets).<br>
\[3] Armv6+ or Linux/Android, except for M-profile architecture such as thumbv6m, thumbv7m, etc.<br>
\[4] Requires `zacas` target feature.<br>
\[5] Requires `scq` target feature and Rust 1.97+.<br>
\[6] Requires `quadword-atomics` target feature (enabled by default on powerpc64le).<br>
\[8] Requires `v9` and `v8plus` target features (both enabled by default on Linux).<br>
\[9] Requires target-cpu M68020 (Linux's default), M68030, M68040, or M68060 (Linux/NetBSD only).<br>
\[10] Requires Rust 1.84+.<br>
\[11] Requires Rust 1.91+.<br>
\[12] Requires Rust 1.95+.<br>
\[13] Requires nightly due to `#![feature(asm_experimental_arch)]`.<br>
<!-- mips32r6/mips64r6: \[7] Requires Release 6 Paired LL/SC family of instructions.<br> -->

See also [Atomic operation overview by architecture](https://github.com/taiki-e/atomic-maybe-uninit/blob/HEAD/src/arch/README.md)
for more information about atomic operations in these architectures.

Feel free to submit an issue if your target is not supported yet.

## Limitations

This crate uses inline assembly to implement atomic operations (this is currently the only sound way to perform atomic operations on uninitialized values), so it is currently not compatible with [Miri](https://github.com/rust-lang/miri/issues/11) and [most kinds of Sanitizers](https://github.com/google/sanitizers/issues/192).

## Related Projects

- [portable-atomic]: Portable atomic types including support for 128-bit atomics, atomic float, etc.
- [atomic-memcpy]: Byte-wise atomic memcpy.
- [asmtest]: A library for tracking generated assemblies.

[asmtest]: https://github.com/taiki-e/asmtest
[atomic-memcpy]: https://github.com/taiki-e/atomic-memcpy
[portable-atomic]: https://github.com/taiki-e/portable-atomic
[undefined-behavior]: https://doc.rust-lang.org/reference/behavior-considered-undefined.html

<!-- tidy:sync-markdown-to-rustdoc:end -->
*/

#![no_std]
#![doc(test(
    no_crate_inject,
    attr(allow(
        dead_code,
        unused_variables,
        clippy::undocumented_unsafe_blocks,
        clippy::unused_trait_names,
    ))
))]
#![warn(
    // Lints that may help when writing public library.
    missing_debug_implementations,
    missing_docs,
    clippy::alloc_instead_of_core,
    clippy::exhaustive_enums,
    clippy::exhaustive_structs,
    clippy::impl_trait_in_params,
    clippy::std_instead_of_alloc,
    clippy::std_instead_of_core,
    clippy::missing_inline_in_public_items,
    // Code outside of cfg(test) shouldn't use float.
    clippy::float_arithmetic,
    // Code outside of cfg(test) shouldn't use code that can panic except for assertions. (overflow also cause panic if overflow check is enabled)
    clippy::arithmetic_side_effects,
)]
#![cfg_attr(atomic_maybe_uninit_no_strict_provenance, allow(unstable_name_collisions))]
#![allow(clippy::inline_always, clippy::unreadable_literal, clippy::used_underscore_items)]
#![cfg_attr(
    all(
        atomic_maybe_uninit_unstable_asm_experimental_arch,
        not(any(
            // These cases currently don't use asm!
            all(target_arch = "sparc", atomic_maybe_uninit_no_stbar),
            all(target_arch = "mips", atomic_maybe_uninit_no_sync),
        )),
    ),
    feature(asm_experimental_arch)
)]

// There are currently no 128-bit or higher builtin targets.
// (Although some of our generic code is written with the future
// addition of 128-bit targets in mind.)
// Note that Rust (and C99) pointers must be at least 16-bit (i.e., 8-bit targets are impossible): https://github.com/rust-lang/rust/pull/49305
#[cfg(not(any(
    target_pointer_width = "16",
    target_pointer_width = "32",
    target_pointer_width = "64",
)))]
compile_error!(
    "atomic-maybe-uninit currently only supports targets with {16,32,64}-bit pointer width; \
     if you need support for others, \
     please submit an issue at <https://github.com/taiki-e/atomic-maybe-uninit>"
);

#[cfg(test)]
extern crate std;

#[macro_use]
mod utils;

#[cfg(test)]
#[macro_use]
mod tests;

pub mod raw;

#[cfg(doc)]
use core::sync::atomic::Ordering::{AcqRel, Acquire, Relaxed, Release, SeqCst};
use core::{
    cell::UnsafeCell,
    fmt,
    mem::{self, MaybeUninit},
    sync::atomic::Ordering,
};

use self::raw::{AtomicCompareExchange, AtomicLoad, AtomicStore, AtomicSwap, Primitive};

// -----------------------------------------------------------------------------
// AtomicMaybeUninit

/// A potentially uninitialized integer type which can be safely shared between threads.
///
/// This type has the same in-memory representation as the underlying
/// value type, `MaybeUninit<T>`.
/// However, the alignment of this type is always equal to its
/// size, even on targets where `MaybeUninit<T>` has a
/// lesser alignment.
#[repr(C)]
pub struct AtomicMaybeUninit<T: Primitive> {
    v: UnsafeCell<MaybeUninit<T>>,
    /// `[T::Align; 0]` ensures alignment is at least that of `T::Align`.
    ///
    /// This is needed because x86's u64 is 4-byte aligned and x86_64's u128 is
    /// 8-byte aligned and atomic operations normally require alignment greater
    /// than or equal to the size.
    _align: [T::Align; 0],
}

impl<T: Primitive> From<MaybeUninit<T>> for AtomicMaybeUninit<T> {
    /// Creates a new atomic value from a potentially uninitialized value.
    #[inline]
    fn from(v: MaybeUninit<T>) -> Self {
        Self::new(v)
    }
}

impl<T: Primitive> From<T> for AtomicMaybeUninit<T> {
    /// Creates a new atomic value from an initialized value.
    #[inline]
    fn from(v: T) -> Self {
        Self::new(MaybeUninit::new(v))
    }
}

impl<T: Primitive> fmt::Debug for AtomicMaybeUninit<T> {
    #[inline] // fmt is not hot path, but #[inline] on fmt seems to still be useful: https://github.com/rust-lang/rust/pull/117727
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str(core::any::type_name::<Self>())
    }
}

// Send is implicitly implemented.
// SAFETY: `T` is `Send` and any data races are prevented by atomic intrinsics.
unsafe impl<T: Primitive> Sync for AtomicMaybeUninit<T> {}

// UnwindSafe is implicitly implemented.
impl<T: Primitive> core::panic::RefUnwindSafe for AtomicMaybeUninit<T> {}

impl<T: Primitive> AtomicMaybeUninit<T> {
    /// Creates a new atomic value from a potentially uninitialized value.
    ///
    /// # Examples
    ///
    /// ```
    /// use std::mem::MaybeUninit;
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// let v = AtomicMaybeUninit::new(MaybeUninit::new(5_i32));
    ///
    /// // Equivalent to:
    /// let v = AtomicMaybeUninit::from(5_i32);
    /// ```
    #[inline]
    #[must_use]
    pub const fn new(v: MaybeUninit<T>) -> Self {
        Self { v: UnsafeCell::new(v), _align: [] }
    }

    /// Creates a new reference to an atomic value from a pointer.
    ///
    /// # Safety
    ///
    /// * `ptr` must be aligned to `align_of::<AtomicMaybeUninit<T>>()` (note that on some platforms this
    ///   can be bigger than `align_of::<MaybeUninit<T>>()`).
    /// * `ptr` must be [valid] for both reads and writes for the whole lifetime `'a`.
    /// * You must adhere to the [Memory model for atomic accesses]. In particular, it is not
    ///   allowed to mix conflicting atomic and non-atomic accesses, or atomic accesses of different
    ///   sizes, without synchronization.
    ///
    /// [valid]: core::ptr#safety
    /// [Memory model for atomic accesses]: core::sync::atomic#memory-model-for-atomic-accesses
    #[inline]
    #[must_use]
    pub const unsafe fn from_ptr<'a>(ptr: *mut MaybeUninit<T>) -> &'a Self {
        // SAFETY: guaranteed by the caller
        unsafe { &*ptr.cast::<Self>().cast_const() }
    }

    const_fn! {
        const_if: #[cfg(not(atomic_maybe_uninit_no_const_mut_refs))];
        /// Returns a mutable reference to the underlying value.
        ///
        /// This is safe because the mutable reference guarantees that no other threads are
        /// concurrently accessing the atomic data.
        ///
        /// This is `const fn` on Rust 1.83+.
        ///
        /// # Examples
        ///
        /// ```
        /// use std::mem::MaybeUninit;
        ///
        /// use atomic_maybe_uninit::AtomicMaybeUninit;
        ///
        /// let mut v = AtomicMaybeUninit::from(5_i32);
        /// unsafe { assert_eq!((*v.get_mut()).assume_init(), 5) }
        /// *v.get_mut() = MaybeUninit::new(10);
        /// unsafe { assert_eq!((*v.get_mut()).assume_init(), 10) }
        /// ```
        #[inline]
        pub const fn get_mut(&mut self) -> &mut MaybeUninit<T> {
            // SAFETY: the mutable reference guarantees unique ownership.
            // (core::cell::UnsafeCell::get_mut requires newer nightly)
            unsafe { &mut *self.as_ptr() }
        }
    }

    /// Consumes the atomic and returns the contained value.
    ///
    /// This is safe because passing `self` by value guarantees that no other threads are
    /// concurrently accessing the atomic data.
    ///
    /// # Examples
    ///
    /// ```
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// let v = AtomicMaybeUninit::from(5_i32);
    /// unsafe { assert_eq!(v.into_inner().assume_init(), 5) }
    /// ```
    #[inline]
    pub const fn into_inner(self) -> MaybeUninit<T> {
        // SAFETY: AtomicMaybeUninit<T> and MaybeUninit<T> have the same size
        // and in-memory representations, so they can be safely transmuted.
        // (Equivalent to UnsafeCell::into_inner which is unstable in const context.)
        unsafe { utils::transmute_copy_by_val::<Self, MaybeUninit<T>>(self) }
    }

    /// Loads a value from the atomic value.
    ///
    /// `load` takes an [`Ordering`] argument which describes the memory ordering of this operation.
    /// Possible values are [`SeqCst`], [`Acquire`] and [`Relaxed`].
    ///
    /// # Panics
    ///
    /// Panics if `order` is [`Release`] or [`AcqRel`].
    ///
    /// # Examples
    ///
    /// ```
    /// use std::sync::atomic::Ordering;
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// let v = AtomicMaybeUninit::from(5_i32);
    /// unsafe { assert_eq!(v.load(Ordering::Relaxed).assume_init(), 5) }
    /// ```
    #[inline]
    #[cfg_attr(debug_assertions, track_caller)]
    pub fn load(&self, order: Ordering) -> MaybeUninit<T>
    where
        T: AtomicLoad,
    {
        utils::assert_load_ordering(order);
        // SAFETY: any data races are prevented by atomic intrinsics, the raw
        // pointer passed in is valid because we got it from a reference,
        // and we've checked the order is valid. Alignment is upheld because
        // `PrimitivePriv`'s safety requirement ensures sufficient alignment
        // of `T::Align`, and we got our `_align` field.
        unsafe { T::atomic_load(self.v.get(), order) }
    }

    /// Stores a value into the atomic value.
    ///
    /// `store` takes an [`Ordering`] argument which describes the memory ordering of this operation.
    ///  Possible values are [`SeqCst`], [`Release`] and [`Relaxed`].
    ///
    /// # Panics
    ///
    /// Panics if `order` is [`Acquire`] or [`AcqRel`].
    ///
    /// # Examples
    ///
    /// ```
    /// use std::{mem::MaybeUninit, sync::atomic::Ordering};
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// let v = AtomicMaybeUninit::from(5_i32);
    /// v.store(MaybeUninit::new(10), Ordering::Relaxed);
    /// unsafe { assert_eq!(v.load(Ordering::Relaxed).assume_init(), 10) }
    /// ```
    #[inline]
    #[cfg_attr(debug_assertions, track_caller)]
    pub fn store(&self, val: MaybeUninit<T>, order: Ordering)
    where
        T: AtomicStore,
    {
        utils::assert_store_ordering(order);
        // SAFETY: any data races are prevented by atomic intrinsics, the raw
        // pointer passed in is valid because we got it from a reference,
        // and we've checked the order is valid. Alignment is upheld because
        // `PrimitivePriv`'s safety requirement ensures sufficient alignment
        // of `T::Align`, and we got our `_align` field.
        unsafe { T::atomic_store(self.v.get(), val, order) }
    }

    /// Stores a value into the atomic value, returning the previous value.
    ///
    /// `swap` takes an [`Ordering`] argument which describes the memory ordering
    /// of this operation. All ordering modes are possible. Note that using
    /// [`Acquire`] makes the store part of this operation [`Relaxed`], and
    /// using [`Release`] makes the load part [`Relaxed`].
    ///
    /// # Examples
    ///
    /// ```
    /// use std::{mem::MaybeUninit, sync::atomic::Ordering};
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// let v = AtomicMaybeUninit::from(5_i32);
    /// unsafe {
    ///     assert_eq!(v.swap(MaybeUninit::new(10), Ordering::Relaxed).assume_init(), 5);
    ///     assert_eq!(v.load(Ordering::Relaxed).assume_init(), 10);
    /// }
    /// ```
    #[inline]
    pub fn swap(&self, val: MaybeUninit<T>, order: Ordering) -> MaybeUninit<T>
    where
        T: AtomicSwap,
    {
        // SAFETY: any data races are prevented by atomic intrinsics and the raw
        // pointer passed in is valid because we got it from a reference.
        // Alignment is upheld because `PrimitivePriv`'s safety requirement
        // ensures sufficient alignment of `T::Align`, and we got our `_align`
        // field.
        unsafe { T::atomic_swap(self.v.get(), val, order) }
    }

    /// Stores a value into the atomic value if the current value is the same as
    /// the `current` value. Here, "the same" is determined using byte-wise
    /// equality, not `PartialEq`.
    ///
    /// The return value is a result indicating whether the new value was written and
    /// containing the previous value. On success this value is guaranteed to be equal to
    /// `current`.
    ///
    /// `compare_exchange` takes two [`Ordering`] arguments to describe the memory
    /// ordering of this operation. `success` describes the required ordering for the
    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    /// `failure` describes the required ordering for the load operation that takes place when
    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    ///
    /// # Panics
    ///
    /// Panics if `failure` is [`Release`], [`AcqRel`].
    ///
    /// # Notes
    ///
    /// Comparison of two values containing uninitialized bytes may fail even if
    /// they are equivalent as Rust's type, because values can be byte-wise
    /// inequal even when they are equal as Rust values.
    ///
    /// For example, the following example could be an infinite loop:
    ///
    /// ```no_run
    /// use std::{
    ///     mem::{self, MaybeUninit},
    ///     sync::atomic::Ordering,
    /// };
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// #[derive(Clone, Copy, PartialEq, Eq)]
    /// #[repr(C, align(4))]
    /// struct Test(u8, u16);
    ///
    /// unsafe {
    ///     let x = mem::transmute::<Test, MaybeUninit<u32>>(Test(0, 0));
    ///     let v = AtomicMaybeUninit::new(x);
    ///     while v
    ///         .compare_exchange(
    ///             mem::transmute::<Test, MaybeUninit<u32>>(Test(0, 0)),
    ///             mem::transmute::<Test, MaybeUninit<u32>>(Test(1, 0)),
    ///             Ordering::AcqRel,
    ///             Ordering::Acquire,
    ///         )
    ///         .is_err()
    ///     {}
    /// }
    /// ```
    ///
    /// To work around this problem, you need to use a helper like the following.
    ///
    /// ```
    /// # use std::{
    /// #     mem::{self, MaybeUninit},
    /// #     sync::atomic::Ordering,
    /// # };
    /// # use atomic_maybe_uninit::AtomicMaybeUninit;
    /// # #[derive(Clone, Copy, PartialEq, Eq)]
    /// # #[repr(C, align(4))]
    /// # struct Test(u8, u16);
    /// // Adapted from https://github.com/crossbeam-rs/crossbeam/blob/crossbeam-utils-0.8.10/crossbeam-utils/src/atomic/atomic_cell.rs#L1081-L1110
    /// unsafe fn atomic_compare_exchange(
    ///     v: &AtomicMaybeUninit<u32>,
    ///     mut current: Test,
    ///     new: Test,
    /// ) -> Result<Test, Test> {
    ///     let mut current_raw = unsafe { mem::transmute::<Test, MaybeUninit<u32>>(current) };
    ///     let new_raw = unsafe { mem::transmute::<Test, MaybeUninit<u32>>(new) };
    ///     loop {
    ///         match v.compare_exchange_weak(current_raw, new_raw, Ordering::AcqRel, Ordering::Acquire)
    ///         {
    ///             Ok(_) => {
    ///                 // The values are byte-wise equal; for `Test` we know this implies they are `PartialEq`-equal.
    ///                 break Ok(current);
    ///             }
    ///             Err(previous_raw) => {
    ///                 let previous = unsafe { mem::transmute::<MaybeUninit<u32>, Test>(previous_raw) };
    ///
    ///                 if !Test::eq(&previous, &current) {
    ///                     break Err(previous);
    ///                 }
    ///
    ///                 // The compare-exchange operation has failed and didn't store `new`. The
    ///                 // failure is either spurious, or `previous` was semantically equal to
    ///                 // `current` but not byte-equal. Let's retry with `previous` as the new
    ///                 // `current`.
    ///                 current = previous;
    ///                 current_raw = previous_raw;
    ///             }
    ///         }
    ///     }
    /// }
    /// # if cfg!(valgrind) { return; }
    ///
    /// unsafe {
    ///     let x = mem::transmute::<Test, MaybeUninit<u32>>(Test(0, 0));
    ///     let v = AtomicMaybeUninit::new(x);
    ///     while atomic_compare_exchange(&v, Test(0, 0), Test(1, 0)).is_err() {}
    /// }
    /// ```
    ///
    /// Also, Valgrind reports "Conditional jump or move depends on uninitialized value(s)"
    /// error if there is such a comparison -- which is correct, that's exactly
    /// what the implementation does, but we are doing this inside inline
    /// assembly so it should be fine. (Effectively we are adding partial
    /// `freeze` capabilities to Rust via inline assembly. This pattern has not
    /// been blessed by the language team, but is also not known to cause any
    /// problems.)
    ///
    /// # Examples
    ///
    /// ```
    /// use std::{mem::MaybeUninit, sync::atomic::Ordering};
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// unsafe {
    ///     let v = AtomicMaybeUninit::from(5_i32);
    ///
    ///     assert_eq!(
    ///         v.compare_exchange(
    ///             MaybeUninit::new(5),
    ///             MaybeUninit::new(10),
    ///             Ordering::Acquire,
    ///             Ordering::Relaxed
    ///         )
    ///         .unwrap()
    ///         .assume_init(),
    ///         5
    ///     );
    ///     assert_eq!(v.load(Ordering::Relaxed).assume_init(), 10);
    ///
    ///     assert_eq!(
    ///         v.compare_exchange(
    ///             MaybeUninit::new(6),
    ///             MaybeUninit::new(12),
    ///             Ordering::SeqCst,
    ///             Ordering::Acquire
    ///         )
    ///         .unwrap_err()
    ///         .assume_init(),
    ///         10
    ///     );
    ///     assert_eq!(v.load(Ordering::Relaxed).assume_init(), 10);
    /// }
    /// ```
    #[doc(alias = "compare_and_swap")]
    #[inline]
    #[cfg_attr(debug_assertions, track_caller)]
    pub fn compare_exchange(
        &self,
        current: MaybeUninit<T>,
        new: MaybeUninit<T>,
        success: Ordering,
        failure: Ordering,
    ) -> Result<MaybeUninit<T>, MaybeUninit<T>>
    where
        T: AtomicCompareExchange,
    {
        utils::assert_compare_exchange_ordering(success, failure);
        // SAFETY: any data races are prevented by atomic intrinsics and the raw
        // pointer passed in is valid because we got it from a reference.
        // Alignment is upheld because `PrimitivePriv`'s safety requirement
        // ensures sufficient alignment of `T::Align`, and we got our `_align`
        // field.
        let (out, ok) =
            unsafe { T::atomic_compare_exchange(self.v.get(), current, new, success, failure) };
        if ok { Ok(out) } else { Err(out) }
    }

    /// Stores a value into the atomic value if the current value is the same as
    /// the `current` value. Here, "the same" is determined using byte-wise
    /// equality, not `PartialEq`.
    ///
    /// This function is allowed to spuriously fail even when the comparison succeeds,
    /// which can result in more efficient code on some platforms. The return value
    /// is a result indicating whether the new value was written and containing
    /// the previous value.
    ///
    /// `compare_exchange_weak` takes two [`Ordering`] arguments to describe the memory
    /// ordering of this operation. `success` describes the required ordering for the
    /// read-modify-write operation that takes place if the comparison with `current` succeeds.
    /// `failure` describes the required ordering for the load operation that takes place when
    /// the comparison fails. Using [`Acquire`] as success ordering makes the store part
    /// of this operation [`Relaxed`], and using [`Release`] makes the successful load
    /// [`Relaxed`]. The failure ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    ///
    /// # Panics
    ///
    /// Panics if `failure` is [`Release`], [`AcqRel`].
    ///
    /// # Notes
    ///
    /// Comparison of two values containing uninitialized bytes may fail even if
    /// they are equivalent as Rust's type, because values can be byte-wise
    /// inequal even when they are equal as Rust values.
    ///
    /// See [`compare_exchange`](Self::compare_exchange) for details.
    ///
    /// # Examples
    ///
    /// ```
    /// use std::{mem::MaybeUninit, sync::atomic::Ordering};
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// let v = AtomicMaybeUninit::from(5_i32);
    ///
    /// unsafe {
    ///     let mut old = v.load(Ordering::Relaxed);
    ///     loop {
    ///         let new = old.assume_init() * 2;
    ///         match v.compare_exchange_weak(
    ///             old,
    ///             MaybeUninit::new(new),
    ///             Ordering::SeqCst,
    ///             Ordering::Relaxed,
    ///         ) {
    ///             Ok(_) => break,
    ///             Err(x) => old = x,
    ///         }
    ///     }
    /// }
    /// ```
    #[doc(alias = "compare_and_swap")]
    #[inline]
    #[cfg_attr(debug_assertions, track_caller)]
    pub fn compare_exchange_weak(
        &self,
        current: MaybeUninit<T>,
        new: MaybeUninit<T>,
        success: Ordering,
        failure: Ordering,
    ) -> Result<MaybeUninit<T>, MaybeUninit<T>>
    where
        T: AtomicCompareExchange,
    {
        utils::assert_compare_exchange_ordering(success, failure);
        // SAFETY: any data races are prevented by atomic intrinsics and the raw
        // pointer passed in is valid because we got it from a reference.
        // Alignment is upheld because `PrimitivePriv`'s safety requirement
        // ensures sufficient alignment of `T::Align`, and we got our `_align`
        // field.
        let (out, ok) = unsafe {
            T::atomic_compare_exchange_weak(self.v.get(), current, new, success, failure)
        };
        if ok { Ok(out) } else { Err(out) }
    }

    /// An alias for [`try_update`](Self::try_update).
    #[inline]
    #[deprecated(note = "renamed to `try_update` for consistency")]
    pub fn fetch_update<F>(
        &self,
        set_order: Ordering,
        fetch_order: Ordering,
        f: F,
    ) -> Result<MaybeUninit<T>, MaybeUninit<T>>
    where
        F: FnMut(MaybeUninit<T>) -> Option<MaybeUninit<T>>,
        T: AtomicCompareExchange,
    {
        self.try_update(set_order, fetch_order, f)
    }

    /// Fetches the value, and applies a function to it that returns an optional
    /// new value. Returns a `Result` of `Ok(previous_value)` if the function returned `Some(_)`, else
    /// `Err(previous_value)`.
    /// See also: [`update`](Self::update).
    ///
    /// Note: This may call the function multiple times if the value has been changed from other threads in
    /// the meantime, as long as the function returns `Some(_)`, but the function will have been applied
    /// only once to the stored value.
    ///
    /// `try_update` takes two [`Ordering`] arguments to describe the memory ordering of this operation.
    /// The first describes the required ordering for when the operation finally succeeds while the second
    /// describes the required ordering for loads. These correspond to the success and failure orderings of
    /// [`compare_exchange`](Self::compare_exchange) respectively.
    ///
    /// Using [`Acquire`] as success ordering makes the store part
    /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
    /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    ///
    /// # Panics
    ///
    /// Panics if `fetch_order` is [`Release`], [`AcqRel`].
    ///
    /// # Considerations
    ///
    /// This method is not magic; it is not provided by the hardware, and does not act like a
    /// critical section or mutex.
    ///
    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem]
    /// if this atomic integer is an index or more generally if knowledge of only the *bitwise value*
    /// of the atomic is not in and of itself sufficient to ensure any required preconditions.
    ///
    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    ///
    /// # Examples
    ///
    /// ```
    /// use std::{mem::MaybeUninit, sync::atomic::Ordering};
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// unsafe {
    ///     let v = AtomicMaybeUninit::from(5_i32);
    ///     assert_eq!(
    ///         v.try_update(Ordering::SeqCst, Ordering::SeqCst, |_| None).unwrap_err().assume_init(),
    ///         5
    ///     );
    ///     assert_eq!(
    ///         v.try_update(Ordering::SeqCst, Ordering::SeqCst, |x| Some(MaybeUninit::new(
    ///             x.assume_init() + 1
    ///         )))
    ///         .unwrap()
    ///         .assume_init(),
    ///         5
    ///     );
    ///     assert_eq!(v.load(Ordering::SeqCst).assume_init(), 6);
    /// }
    /// ```
    #[allow(clippy::impl_trait_in_params)] // Align to core::sync::atomic
    #[inline]
    pub fn try_update(
        &self,
        set_order: Ordering,
        fetch_order: Ordering,
        mut f: impl FnMut(MaybeUninit<T>) -> Option<MaybeUninit<T>>,
    ) -> Result<MaybeUninit<T>, MaybeUninit<T>>
    where
        T: AtomicCompareExchange,
    {
        let mut prev = self.load(fetch_order);
        while let Some(next) = f(prev) {
            match self.compare_exchange_weak(prev, next, set_order, fetch_order) {
                x @ Ok(_) => return x,
                Err(next_prev) => prev = next_prev,
            }
        }
        Err(prev)
    }

    /// Fetches the value, applies a function to it that it return a new value.
    /// The new value is stored and the old value is returned.
    /// See also: [`try_update`](Self::try_update).
    ///
    /// Note: This may call the function multiple times if the value has been changed from other threads in
    /// the meantime, but the function will have been applied only once to the stored value.
    ///
    /// `update` takes two [`Ordering`] arguments to describe the memory ordering of this operation.
    /// The first describes the required ordering for when the operation finally succeeds while the second
    /// describes the required ordering for loads. These correspond to the success and failure orderings of
    /// [`compare_exchange`](Self::compare_exchange) respectively.
    ///
    /// Using [`Acquire`] as success ordering makes the store part
    /// of this operation [`Relaxed`], and using [`Release`] makes the final successful load
    /// [`Relaxed`]. The (failed) load ordering can only be [`SeqCst`], [`Acquire`] or [`Relaxed`].
    ///
    /// # Panics
    ///
    /// Panics if `fetch_order` is [`Release`], [`AcqRel`].
    ///
    /// # Considerations
    ///
    /// [CAS operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    /// This method is not magic; it is not provided by the hardware, and does not act like a
    /// critical section or mutex.
    ///
    /// It is implemented on top of an atomic [compare-and-swap operation], and thus is subject to
    /// the usual drawbacks of CAS operations. In particular, be careful of the [ABA problem]
    /// if this atomic integer is an index or more generally if knowledge of only the *bitwise value*
    /// of the atomic is not in and of itself sufficient to ensure any required preconditions.
    ///
    /// [ABA Problem]: https://en.wikipedia.org/wiki/ABA_problem
    /// [compare-and-swap operation]: https://en.wikipedia.org/wiki/Compare-and-swap
    ///
    /// # Examples
    ///
    /// ```
    /// use std::{mem::MaybeUninit, sync::atomic::Ordering};
    ///
    /// use atomic_maybe_uninit::AtomicMaybeUninit;
    ///
    /// unsafe {
    ///     let v = AtomicMaybeUninit::from(5_i32);
    ///     assert_eq!(
    ///         v.update(Ordering::SeqCst, Ordering::SeqCst, |x| MaybeUninit::new(x.assume_init() + 1))
    ///             .assume_init(),
    ///         5
    ///     );
    ///     assert_eq!(
    ///         v.update(Ordering::SeqCst, Ordering::SeqCst, |x| MaybeUninit::new(x.assume_init() + 1))
    ///             .assume_init(),
    ///         6
    ///     );
    ///     assert_eq!(v.load(Ordering::SeqCst).assume_init(), 7);
    /// }
    /// ```
    #[allow(clippy::impl_trait_in_params)] // Align to core::sync::atomic
    #[inline]
    pub fn update(
        &self,
        set_order: Ordering,
        fetch_order: Ordering,
        mut f: impl FnMut(MaybeUninit<T>) -> MaybeUninit<T>,
    ) -> MaybeUninit<T>
    where
        T: AtomicCompareExchange,
    {
        let mut prev = self.load(fetch_order);
        loop {
            match self.compare_exchange_weak(prev, f(prev), set_order, fetch_order) {
                Ok(x) => break x,
                Err(next_prev) => prev = next_prev,
            }
        }
    }

    /// Returns a mutable pointer to the underlying value.
    ///
    /// Doing non-atomic reads and writes on the resulting value can be a data race.
    /// This method is mostly useful for FFI, where the function signature may use
    /// `*mut T` instead of `&AtomicMaybeUninit<T>`.
    ///
    /// Returning an `*mut` pointer from a shared reference to this atomic is safe because the
    /// atomic types work with interior mutability. All modifications of an atomic change the value
    /// through a shared reference, and can do so safely as long as they use atomic operations. Any
    /// use of the returned raw pointer requires an `unsafe` block and still has to uphold the
    /// requirements of the [memory model].
    ///
    /// [memory model]: core::sync::atomic#memory-model-for-atomic-accesses
    #[inline]
    pub const fn as_ptr(&self) -> *mut MaybeUninit<T> {
        self.v.get()
    }
}

macro_rules! int {
    ($($ty:ident),* => $align:ident) => {$(
        impl raw::Primitive for $ty {}
        const _: () = {
            assert!(mem::size_of::<AtomicMaybeUninit<$ty>>() == mem::size_of::<$ty>());
            assert!(mem::align_of::<AtomicMaybeUninit<$ty>>() >= mem::size_of::<$ty>());
        };
        // SAFETY: the static assertion above ensures safety requirement.
        unsafe impl private::PrimitivePriv for $ty {
            type Align = private::$align;
        }
        impl AtomicMaybeUninit<$ty> {
            /// Creates a new atomic value from a potentially uninitialized value.
            #[inline]
            #[must_use]
            // TODO(semver): remove in the next breaking release.
            #[deprecated(
                since = "0.3.10",
                note = "use `new` instead because it is now always `const fn`"
            )]
            pub const fn const_new(v: MaybeUninit<$ty>) -> Self {
                Self { v: UnsafeCell::new(v), _align: [] }
            }
        }
    )*};
}
int!(i8, u8 => Align1);
int!(i16, u16 => Align2);
int!(i32, u32 => Align4);
int!(i64, u64 => Align8);
int!(i128, u128 => Align16);
int!(isize, usize => AlignPtr);

#[cfg(target_pointer_width = "16")]
pub use {cfg_has_atomic_16 as cfg_has_atomic_ptr, cfg_no_atomic_16 as cfg_no_atomic_ptr};
#[cfg(target_pointer_width = "32")]
pub use {cfg_has_atomic_32 as cfg_has_atomic_ptr, cfg_no_atomic_32 as cfg_no_atomic_ptr};
#[cfg(target_pointer_width = "64")]
pub use {cfg_has_atomic_64 as cfg_has_atomic_ptr, cfg_no_atomic_64 as cfg_no_atomic_ptr};
#[cfg(target_pointer_width = "128")]
pub use {cfg_has_atomic_128 as cfg_has_atomic_ptr, cfg_no_atomic_128 as cfg_no_atomic_ptr};

// -----------------------------------------------------------------------------
// Internals

#[cfg_attr(
    any(target_arch = "aarch64", all(target_arch = "arm64ec", not(atomic_maybe_uninit_no_asm))),
    path = "arch/aarch64.rs"
)]
#[cfg_attr(
    all(
        target_arch = "arm",
        // Pre-v6 Arm has no Data Memory Barrier (DMB) operation, so we cannot implement non-relaxed atomics.
        // However, Linux kernel provides helpers for it, so we can provide it on Linux/Android.
        any(
            target_feature = "v6",
            atomic_maybe_uninit_target_feature = "v6",
            target_os = "linux",
            target_os = "android",
        ),
        // Use armv8.rs for Armv8+.
        not(any(
            target_feature = "v8",
            target_feature = "acquire-release",
            atomic_maybe_uninit_target_feature = "acquire-release",
        )),
    ),
    path = "arch/arm.rs"
)]
#[cfg_attr(
    all(
        target_arch = "arm",
        // Use arm.rs for pre-v8 Arm.
        any(
            target_feature = "v8",
            target_feature = "acquire-release",
            atomic_maybe_uninit_target_feature = "acquire-release",
        ),
    ),
    path = "arch/armv8.rs"
)]
#[cfg_attr(
    all(target_arch = "avr", atomic_maybe_uninit_unstable_asm_experimental_arch),
    path = "arch/avr.rs"
)]
#[cfg_attr(
    all(target_arch = "csky", atomic_maybe_uninit_unstable_asm_experimental_arch),
    path = "arch/csky.rs"
)]
#[cfg_attr(
    all(target_arch = "hexagon", atomic_maybe_uninit_unstable_asm_experimental_arch),
    path = "arch/hexagon.rs"
)]
#[cfg_attr(
    any(
        all(target_arch = "loongarch32", not(atomic_maybe_uninit_no_asm)),
        target_arch = "loongarch64",
    ),
    path = "arch/loongarch.rs"
)]
#[cfg_attr(
    all(target_arch = "m68k", atomic_maybe_uninit_unstable_asm_experimental_arch),
    path = "arch/m68k.rs"
)]
#[cfg_attr(
    all(
        any(
            // MIPS-I has no SYNC, so we cannot implement non-relaxed atomics.
            all(target_arch = "mips", not(atomic_maybe_uninit_no_sync)),
            target_arch = "mips32r6",
            target_arch = "mips64",
            target_arch = "mips64r6",
        ),
        atomic_maybe_uninit_unstable_asm_experimental_arch,
    ),
    path = "arch/mips.rs"
)]
#[cfg_attr(
    all(target_arch = "msp430", atomic_maybe_uninit_unstable_asm_experimental_arch),
    path = "arch/msp430.rs"
)]
#[cfg_attr(
    all(any(target_arch = "powerpc", target_arch = "powerpc64"), not(atomic_maybe_uninit_no_asm)),
    path = "arch/powerpc.rs"
)]
#[cfg_attr(any(target_arch = "riscv32", target_arch = "riscv64"), path = "arch/riscv.rs")]
#[cfg_attr(all(target_arch = "s390x", not(atomic_maybe_uninit_no_asm)), path = "arch/s390x.rs")]
#[cfg_attr(
    all(
        any(
            // SPARC-V7 has no STBAR, so we cannot implement non-relaxed atomics.
            all(target_arch = "sparc", not(atomic_maybe_uninit_no_stbar)),
            target_arch = "sparc64",
        ),
        atomic_maybe_uninit_unstable_asm_experimental_arch,
    ),
    path = "arch/sparc.rs"
)]
#[cfg_attr(any(target_arch = "x86", target_arch = "x86_64"), path = "arch/x86.rs")]
#[cfg_attr(
    all(target_arch = "xtensa", atomic_maybe_uninit_unstable_asm_experimental_arch),
    path = "arch/xtensa.rs"
)]
#[allow(missing_docs)] // For cfg_* macros.
mod arch;

mod private {
    #![allow(missing_debug_implementations)]

    use core::panic::{RefUnwindSafe, UnwindSafe};

    /// This trait is private and cannot be implemented for types outside of `atomic-maybe-uninit`.
    ///
    /// # Safety
    ///
    /// The implementer must guarantee that `align_of::<Self::Align>() == size_of::<Self>()`.
    // Auto traits are needed to better docs.
    #[allow(unknown_lints, unnameable_types)] // Not public API. unnameable_types is available on Rust 1.79+
    pub unsafe trait PrimitivePriv:
        Copy + Send + Sync + Unpin + UnwindSafe + RefUnwindSafe
    {
        // See _align field of AtomicMaybeUninit.
        type Align: Send + Sync + Unpin + UnwindSafe + RefUnwindSafe;
    }

    #[repr(align(1))]
    #[allow(unknown_lints, unnameable_types)] // Not public API. unnameable_types is available on Rust 1.79+
    pub struct Align1(#[allow(dead_code)] u8);
    #[repr(align(2))]
    #[allow(unknown_lints, unnameable_types)] // Not public API. unnameable_types is available on Rust 1.79+
    pub struct Align2(#[allow(dead_code)] u16);
    #[repr(align(4))]
    #[allow(unknown_lints, unnameable_types)] // Not public API. unnameable_types is available on Rust 1.79+
    pub struct Align4(#[allow(dead_code)] u32);
    #[repr(align(8))]
    #[allow(unknown_lints, unnameable_types)] // Not public API. unnameable_types is available on Rust 1.79+
    pub struct Align8(#[allow(dead_code)] u64);
    #[repr(align(16))]
    #[allow(unknown_lints, unnameable_types)] // Not public API. unnameable_types is available on Rust 1.79+
    pub struct Align16(#[allow(dead_code)] u128);
    #[cfg(target_pointer_width = "16")]
    pub(crate) type AlignPtr = Align2;
    #[cfg(target_pointer_width = "32")]
    pub(crate) type AlignPtr = Align4;
    #[cfg(target_pointer_width = "64")]
    pub(crate) type AlignPtr = Align8;
    #[cfg(target_pointer_width = "128")]
    pub(crate) type AlignPtr = Align16;

    // Check that all cfg_ macros work.
    use crate::{
        AtomicMaybeUninit, cfg_has_atomic_8, cfg_has_atomic_16, cfg_has_atomic_32,
        cfg_has_atomic_64, cfg_has_atomic_128, cfg_has_atomic_cas, cfg_has_atomic_ptr,
        cfg_no_atomic_8, cfg_no_atomic_16, cfg_no_atomic_32, cfg_no_atomic_64, cfg_no_atomic_128,
        cfg_no_atomic_cas, cfg_no_atomic_ptr,
    };
    // TODO: make these type aliases public?
    cfg_has_atomic_8! {
        type _AtomicMaybeUninitI8 = AtomicMaybeUninit<i8>;
        type _AtomicMaybeUninitU8 = AtomicMaybeUninit<u8>;
    }
    cfg_no_atomic_8! {
        type _AtomicMaybeUninitI8 = AtomicMaybeUninit<i8>;
        type _AtomicMaybeUninitU8 = AtomicMaybeUninit<u8>;
    }
    cfg_has_atomic_16! {
        type _AtomicMaybeUninitI16 = AtomicMaybeUninit<i16>;
        type _AtomicMaybeUninitU16 = AtomicMaybeUninit<u16>;
    }
    cfg_no_atomic_16! {
        type _AtomicMaybeUninitI16 = AtomicMaybeUninit<i16>;
        type _AtomicMaybeUninitU16 = AtomicMaybeUninit<u16>;
    }
    cfg_has_atomic_32! {
        type _AtomicMaybeUninitI32 = AtomicMaybeUninit<i32>;
        type _AtomicMaybeUninitU32 = AtomicMaybeUninit<u32>;
    }
    cfg_no_atomic_32! {
        type _AtomicMaybeUninitI32 = AtomicMaybeUninit<i32>;
        type _AtomicMaybeUninitU32 = AtomicMaybeUninit<u32>;
    }
    cfg_has_atomic_64! {
        type _AtomicMaybeUninitI64 = AtomicMaybeUninit<i64>;
        type _AtomicMaybeUninitU64 = AtomicMaybeUninit<u64>;
    }
    cfg_no_atomic_64! {
        type _AtomicMaybeUninitI64 = AtomicMaybeUninit<i64>;
        type _AtomicMaybeUninitU64 = AtomicMaybeUninit<u64>;
    }
    cfg_has_atomic_128! {
        type _AtomicMaybeUninitI128 = AtomicMaybeUninit<i128>;
        type _AtomicMaybeUninitU128 = AtomicMaybeUninit<u128>;
    }
    cfg_no_atomic_128! {
        type _AtomicMaybeUninitI128 = AtomicMaybeUninit<i128>;
        type _AtomicMaybeUninitU128 = AtomicMaybeUninit<u128>;
    }
    cfg_has_atomic_ptr! {
        type _AtomicMaybeUninitIsize = AtomicMaybeUninit<isize>;
        type _AtomicMaybeUninitUsize = AtomicMaybeUninit<usize>;
    }
    cfg_no_atomic_ptr! {
        type _AtomicMaybeUninitIsize = AtomicMaybeUninit<isize>;
        type _AtomicMaybeUninitUsize = AtomicMaybeUninit<usize>;
    }
    cfg_has_atomic_cas! {
        type __AtomicMaybeUninitIsize = AtomicMaybeUninit<isize>;
        type __AtomicMaybeUninitUsize = AtomicMaybeUninit<usize>;
    }
    cfg_no_atomic_cas! {
        type __AtomicMaybeUninitIsize = AtomicMaybeUninit<isize>;
        type __AtomicMaybeUninitUsize = AtomicMaybeUninit<usize>;
    }
}