Struct SveArch 

pub struct SveArch;

AArch64 SVE architecture ZST marker.

The current stable implementation uses the same real lane-emulated kernel family as other non-native (architecture, scalar) pairs. Native SVE intrinsics can replace this implementation once stable Rust exposes the required scalable-vector types.

Trait Implementations

impl Clone for SveArch

fn clone(&self) -> SveArch

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for SveArch

impl Debug for SveArch

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Eq for SveArch

impl PartialEq for SveArch

fn eq(&self, other: &SveArch) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl SimdArch for SveArch

const NAME: &'static str = "sve-emulated"

Human-readable ISA name ("avx2", "avx512", "neon", "scalar").

const REGISTER_WIDTH_BITS: u32 = 512

Width of vector registers in bits.

const ISA_FAMILY: IsaFamily = IsaFamily::AArch64

ISA family for this architecture.

const FMA_THROUGHPUT_HINT: u32 = 4

Suggested TILE_M value for tiled_dot to saturate FMA throughput.

fn is_runtime_supported() -> bool

Returns true when the current host may execute this architecture’s native instructions from safe wrappers.

impl SimdKernel<f32> for SveArch

const LANE_COUNT: usize = 16

Number of primitive elements of type T in one Vector.

const UNROLL_FACTOR: usize = 4

Loop unrolling register accumulation factor to break loop-carried dependency chains.

type Vector = [f32; 16]

The underlying raw register/vector type for this architecture and element type.

type Mask = [bool; 16]

Hardware-native mask type.

type IndexVector = [i32; 16]

Integer index vector for gather operations.

unsafe fn load_aligned(ptr: *const f32) -> <SveArch as SimdKernel<f32>>::Vector

Load a vector from an aligned pointer.

unsafe fn load_unaligned( ptr: *const f32, ) -> <SveArch as SimdKernel<f32>>::Vector

Load a vector from an unaligned pointer.

unsafe fn store_aligned( ptr: *mut f32, val: <SveArch as SimdKernel<f32>>::Vector, )

Store a vector to an aligned pointer.

unsafe fn store_unaligned( ptr: *mut f32, val: <SveArch as SimdKernel<f32>>::Vector, )

Store a vector to an unaligned pointer.

unsafe fn add( a: <SveArch as SimdKernel<f32>>::Vector, b: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Vector

Elementwise addition: a + b.

unsafe fn mul( a: <SveArch as SimdKernel<f32>>::Vector, b: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Vector

Elementwise multiplication: a * b.

unsafe fn sub( a: <SveArch as SimdKernel<f32>>::Vector, b: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Vector

Elementwise subtraction: a - b.

unsafe fn neg( a: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Vector

Elementwise negate: -a.

unsafe fn fmadd( a: <SveArch as SimdKernel<f32>>::Vector, b: <SveArch as SimdKernel<f32>>::Vector, c: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Vector

Fused multiply-add: (a * b) + c.

unsafe fn sum_reduce(v: <SveArch as SimdKernel<f32>>::Vector) -> f32

Horizontal sum of all lanes.

unsafe fn compress( src: <SveArch as SimdKernel<f32>>::Vector, mask: <SveArch as SimdKernel<f32>>::Mask, ) -> <SveArch as SimdKernel<f32>>::Vector

Compress: pack selected lanes (where mask[i]=1) into the low lanes of the result.

unsafe fn expand( src: <SveArch as SimdKernel<f32>>::Vector, mask: <SveArch as SimdKernel<f32>>::Mask, fill: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Vector

Expand: scatter the low lanes of src into result positions where mask[i]=1.

unsafe fn gather( base: *const f32, indices: <SveArch as SimdKernel<f32>>::IndexVector, ) -> <SveArch as SimdKernel<f32>>::Vector

Gather: load LANE_COUNT elements at base + indices[i] for each lane i.

unsafe fn gather_masked( base: *const f32, indices: <SveArch as SimdKernel<f32>>::IndexVector, mask: <SveArch as SimdKernel<f32>>::Mask, src: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Vector

Masked gather: gather active lanes; inactive lanes take value from src.

unsafe fn mask_from_bools(bits: &[bool]) -> <SveArch as SimdKernel<f32>>::Mask

Construct a mask from a slice of booleans (length must equal LANE_COUNT).

unsafe fn leading_k_mask(k: usize) -> <SveArch as SimdKernel<f32>>::Mask

Construct a mask with the first k lanes active and the rest inactive.

unsafe fn zero() -> <SveArch as SimdKernel<f32>>::Vector

Set all lanes to zero.

unsafe fn splat(val: f32) -> <SveArch as SimdKernel<f32>>::Vector

Broadcast a scalar value to all lanes.

unsafe fn mask_to_bitmask(mask: <SveArch as SimdKernel<f32>>::Mask) -> u64

Convert the native mask back to a raw u64 bitmask.

unsafe fn mask_to_vector( mask: <SveArch as SimdKernel<f32>>::Mask, ) -> <SveArch as SimdKernel<f32>>::Vector

Convert the native mask back to a vector register where active lanes are set to T::ALL_ONES and inactive lanes to T::ZERO.

unsafe fn vector_to_mask( v: <SveArch as SimdKernel<f32>>::Vector, ) -> <SveArch as SimdKernel<f32>>::Mask

Convert a comparison-result vector into the native mask, the inverse of SimdKernel::mask_to_vector.

const LANE_BOUND_CHECK: () = _

Compile-time guard that LANE_COUNT fits the fixed MAX_SIMD_LANES scalar-fallback stack buffers. Referencing this const in the buffer-using default methods forces the assertion to be evaluated for each concrete backend at monomorphization, turning a would-be silent stack-buffer overflow into a compile error.

const SUPPORTS_NT_STORE: bool = false

Whether this backend provides a non-temporal (cache-bypassing) store via store_streaming. Backends leaving this false keep the regular store default; callers gate the streaming path on this const so it is a compile-time branch, dead-code-eliminated where unsupported.

unsafe fn store_streaming(ptr: *mut T, val: Self::Vector)

Store a vector with a non-temporal (streaming) hint that bypasses the cache, avoiding the read-for-ownership traffic a normal write-allocate pays for write-only data larger than the last-level cache (measured 1.71× on out-of-LLC AVX2 f32 elementwise writes; see streaming_bench).

fn stream_write_barrier()

Fence ordering this backend’s non-temporal stores before subsequent reads. No-op by default (only meaningful where store_streaming is a weakly ordered non-temporal store).

unsafe fn masked_load_unaligned( ptr: *const T, mask: Self::Mask, src: Self::Vector, ) -> Self::Vector

Masked load: active lanes loaded from ptr, inactive lanes taken from src.

unsafe fn masked_store_unaligned( ptr: *mut T, mask: Self::Mask, val: Self::Vector, )

Masked store: active lanes written to ptr, inactive lanes left unchanged.

unsafe fn masked_add( a: Self::Vector, b: Self::Vector, mask: Self::Mask, src: Self::Vector, ) -> Self::Vector

Masked elementwise add: active lanes compute a + b, inactive lanes yield src.

unsafe fn masked_mul( a: Self::Vector, b: Self::Vector, mask: Self::Mask, src: Self::Vector, ) -> Self::Vector

Masked elementwise multiply: active lanes compute a * b, inactive lanes yield src.

unsafe fn masked_fmadd( a: Self::Vector, b: Self::Vector, c: Self::Vector, mask: Self::Mask, ) -> Self::Vector

Masked fused multiply-add: active lanes compute (a * b) + c, inactive lanes retain c.

unsafe fn masked_sum_reduce(v: Self::Vector, mask: Self::Mask) -> T

Masked horizontal sum: only lanes where mask[i]=1 contribute.

unsafe fn mask_from_bitmask(bm: u64) -> Self::Mask

Convert a raw u64 bitmask to the architecture-native mask type.

unsafe fn scan_vector<Op, SMode>(v: Self::Vector, carry: T) -> (Self::Vector, T)

where Op: ScanOp<T>, SMode: ScanMode,

Perform an intra-vector prefix scan (inclusive or exclusive) of the vector, using the specified ScanOp strategy and starting carry value. Returns the scanned vector and the final carry value.

unsafe fn div(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise division: a / b.

unsafe fn bitand(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise bitwise AND: a & b.

unsafe fn bitor(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise bitwise OR: a | b.

unsafe fn bitxor(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise bitwise XOR: a ^ b.

unsafe fn abs(a: Self::Vector) -> Self::Vector

Elementwise absolute value.

unsafe fn min(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise minimum of a and b.

unsafe fn max(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise maximum of a and b.

unsafe fn sqrt(a: Self::Vector) -> Self::Vector

Elementwise square root.

unsafe fn recip_sqrt(a: Self::Vector) -> Self::Vector

Elementwise reciprocal square root, 1/√x, to full T precision (~1 ulp).

unsafe fn cmp_eq(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise equal: a == b.

unsafe fn cmp_ne(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise not equal: a != b.

unsafe fn cmp_lt(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise less than: a < b.

unsafe fn cmp_le(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise less than or equal: a <= b.

unsafe fn cmp_gt(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise greater than: a > b.

unsafe fn cmp_ge(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise greater than or equal: a >= b.

unsafe fn blend( mask: Self::Vector, true_val: Self::Vector, false_val: Self::Vector, ) -> Self::Vector

Elementwise blend: select lanes from true_val where the sign bit of mask is set, and from false_val otherwise.

unsafe fn bitnot(a: Self::Vector) -> Self::Vector

Elementwise bitwise NOT: !a.

unsafe fn min_reduce(v: Self::Vector) -> T

Horizontal minimum across all lanes.

unsafe fn max_reduce(v: Self::Vector) -> T

Horizontal maximum across all lanes.

unsafe fn popcount(a: Self::Vector) -> Self::Vector

Elementwise population count (number of set bits).

unsafe fn horizontal_bitwise_and(v: Self::Vector) -> T

Horizontal bitwise AND across all lanes.

unsafe fn horizontal_bitwise_or(v: Self::Vector) -> T

Horizontal bitwise OR across all lanes.

unsafe fn horizontal_bitwise_xor(v: Self::Vector) -> T

Horizontal bitwise XOR across all lanes.

unsafe fn swap_adjacent(v: Self::Vector) -> Self::Vector

Swap each adjacent lane pair: [a0, a1, a2, a3, ...] -> [a1, a0, a3, a2, ...].

unsafe fn dup_even(v: Self::Vector) -> Self::Vector

Duplicate even lanes into odd lanes: [a0, a1, a2, a3, ...] -> [a0, a0, a2, a2, ...].

unsafe fn dup_odd(v: Self::Vector) -> Self::Vector

Duplicate odd lanes into even lanes: [a0, a1, a2, a3, ...] -> [a1, a1, a3, a3, ...].

unsafe fn fmaddsub( a: Self::Vector, b: Self::Vector, c: Self::Vector, ) -> Self::Vector

Alternating fused multiply: even lanes a*b - c, odd lanes a*b + c.

unsafe fn fmsubadd( a: Self::Vector, b: Self::Vector, c: Self::Vector, ) -> Self::Vector

Alternating fused multiply: even lanes a*b + c, odd lanes a*b - c.

impl SimdKernel<f64> for SveArch

const LANE_COUNT: usize = 8

Number of primitive elements of type T in one Vector.

const UNROLL_FACTOR: usize = 4

Loop unrolling register accumulation factor to break loop-carried dependency chains.

type Vector = [f64; 8]

The underlying raw register/vector type for this architecture and element type.

type Mask = [bool; 8]

Hardware-native mask type.

type IndexVector = [i32; 8]

Integer index vector for gather operations.

unsafe fn load_aligned(ptr: *const f64) -> <SveArch as SimdKernel<f64>>::Vector

Load a vector from an aligned pointer.

unsafe fn load_unaligned( ptr: *const f64, ) -> <SveArch as SimdKernel<f64>>::Vector

Load a vector from an unaligned pointer.

unsafe fn store_aligned( ptr: *mut f64, val: <SveArch as SimdKernel<f64>>::Vector, )

Store a vector to an aligned pointer.

unsafe fn store_unaligned( ptr: *mut f64, val: <SveArch as SimdKernel<f64>>::Vector, )

Store a vector to an unaligned pointer.

unsafe fn add( a: <SveArch as SimdKernel<f64>>::Vector, b: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Vector

Elementwise addition: a + b.

unsafe fn mul( a: <SveArch as SimdKernel<f64>>::Vector, b: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Vector

Elementwise multiplication: a * b.

unsafe fn sub( a: <SveArch as SimdKernel<f64>>::Vector, b: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Vector

Elementwise subtraction: a - b.

unsafe fn neg( a: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Vector

Elementwise negate: -a.

unsafe fn fmadd( a: <SveArch as SimdKernel<f64>>::Vector, b: <SveArch as SimdKernel<f64>>::Vector, c: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Vector

Fused multiply-add: (a * b) + c.

unsafe fn sum_reduce(v: <SveArch as SimdKernel<f64>>::Vector) -> f64

Horizontal sum of all lanes.

unsafe fn compress( src: <SveArch as SimdKernel<f64>>::Vector, mask: <SveArch as SimdKernel<f64>>::Mask, ) -> <SveArch as SimdKernel<f64>>::Vector

Compress: pack selected lanes (where mask[i]=1) into the low lanes of the result.

unsafe fn expand( src: <SveArch as SimdKernel<f64>>::Vector, mask: <SveArch as SimdKernel<f64>>::Mask, fill: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Vector

Expand: scatter the low lanes of src into result positions where mask[i]=1.

unsafe fn gather( base: *const f64, indices: <SveArch as SimdKernel<f64>>::IndexVector, ) -> <SveArch as SimdKernel<f64>>::Vector

Gather: load LANE_COUNT elements at base + indices[i] for each lane i.

unsafe fn gather_masked( base: *const f64, indices: <SveArch as SimdKernel<f64>>::IndexVector, mask: <SveArch as SimdKernel<f64>>::Mask, src: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Vector

Masked gather: gather active lanes; inactive lanes take value from src.

unsafe fn mask_from_bools(bits: &[bool]) -> <SveArch as SimdKernel<f64>>::Mask

Construct a mask from a slice of booleans (length must equal LANE_COUNT).

unsafe fn leading_k_mask(k: usize) -> <SveArch as SimdKernel<f64>>::Mask

Construct a mask with the first k lanes active and the rest inactive.

unsafe fn zero() -> <SveArch as SimdKernel<f64>>::Vector

Set all lanes to zero.

unsafe fn splat(val: f64) -> <SveArch as SimdKernel<f64>>::Vector

Broadcast a scalar value to all lanes.

unsafe fn mask_to_bitmask(mask: <SveArch as SimdKernel<f64>>::Mask) -> u64

Convert the native mask back to a raw u64 bitmask.

unsafe fn mask_to_vector( mask: <SveArch as SimdKernel<f64>>::Mask, ) -> <SveArch as SimdKernel<f64>>::Vector

Convert the native mask back to a vector register where active lanes are set to T::ALL_ONES and inactive lanes to T::ZERO.

unsafe fn vector_to_mask( v: <SveArch as SimdKernel<f64>>::Vector, ) -> <SveArch as SimdKernel<f64>>::Mask

Convert a comparison-result vector into the native mask, the inverse of SimdKernel::mask_to_vector.

const LANE_BOUND_CHECK: () = _

Compile-time guard that LANE_COUNT fits the fixed MAX_SIMD_LANES scalar-fallback stack buffers. Referencing this const in the buffer-using default methods forces the assertion to be evaluated for each concrete backend at monomorphization, turning a would-be silent stack-buffer overflow into a compile error.

const SUPPORTS_NT_STORE: bool = false

Whether this backend provides a non-temporal (cache-bypassing) store via store_streaming. Backends leaving this false keep the regular store default; callers gate the streaming path on this const so it is a compile-time branch, dead-code-eliminated where unsupported.

unsafe fn store_streaming(ptr: *mut T, val: Self::Vector)

Store a vector with a non-temporal (streaming) hint that bypasses the cache, avoiding the read-for-ownership traffic a normal write-allocate pays for write-only data larger than the last-level cache (measured 1.71× on out-of-LLC AVX2 f32 elementwise writes; see streaming_bench).

fn stream_write_barrier()

Fence ordering this backend’s non-temporal stores before subsequent reads. No-op by default (only meaningful where store_streaming is a weakly ordered non-temporal store).

unsafe fn masked_load_unaligned( ptr: *const T, mask: Self::Mask, src: Self::Vector, ) -> Self::Vector

Masked load: active lanes loaded from ptr, inactive lanes taken from src.

unsafe fn masked_store_unaligned( ptr: *mut T, mask: Self::Mask, val: Self::Vector, )

Masked store: active lanes written to ptr, inactive lanes left unchanged.

unsafe fn masked_add( a: Self::Vector, b: Self::Vector, mask: Self::Mask, src: Self::Vector, ) -> Self::Vector

Masked elementwise add: active lanes compute a + b, inactive lanes yield src.

unsafe fn masked_mul( a: Self::Vector, b: Self::Vector, mask: Self::Mask, src: Self::Vector, ) -> Self::Vector

Masked elementwise multiply: active lanes compute a * b, inactive lanes yield src.

unsafe fn masked_fmadd( a: Self::Vector, b: Self::Vector, c: Self::Vector, mask: Self::Mask, ) -> Self::Vector

Masked fused multiply-add: active lanes compute (a * b) + c, inactive lanes retain c.

unsafe fn masked_sum_reduce(v: Self::Vector, mask: Self::Mask) -> T

Masked horizontal sum: only lanes where mask[i]=1 contribute.

unsafe fn mask_from_bitmask(bm: u64) -> Self::Mask

Convert a raw u64 bitmask to the architecture-native mask type.

unsafe fn scan_vector<Op, SMode>(v: Self::Vector, carry: T) -> (Self::Vector, T)

where Op: ScanOp<T>, SMode: ScanMode,

Perform an intra-vector prefix scan (inclusive or exclusive) of the vector, using the specified ScanOp strategy and starting carry value. Returns the scanned vector and the final carry value.

unsafe fn div(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise division: a / b.

unsafe fn bitand(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise bitwise AND: a & b.

unsafe fn bitor(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise bitwise OR: a | b.

unsafe fn bitxor(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise bitwise XOR: a ^ b.

unsafe fn abs(a: Self::Vector) -> Self::Vector

Elementwise absolute value.

unsafe fn min(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise minimum of a and b.

unsafe fn max(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise maximum of a and b.

unsafe fn sqrt(a: Self::Vector) -> Self::Vector

Elementwise square root.

unsafe fn recip_sqrt(a: Self::Vector) -> Self::Vector

Elementwise reciprocal square root, 1/√x, to full T precision (~1 ulp).

unsafe fn cmp_eq(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise equal: a == b.

unsafe fn cmp_ne(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise not equal: a != b.

unsafe fn cmp_lt(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise less than: a < b.

unsafe fn cmp_le(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise less than or equal: a <= b.

unsafe fn cmp_gt(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise greater than: a > b.

unsafe fn cmp_ge(a: Self::Vector, b: Self::Vector) -> Self::Vector

Elementwise greater than or equal: a >= b.

unsafe fn blend( mask: Self::Vector, true_val: Self::Vector, false_val: Self::Vector, ) -> Self::Vector

Elementwise blend: select lanes from true_val where the sign bit of mask is set, and from false_val otherwise.

unsafe fn bitnot(a: Self::Vector) -> Self::Vector

Elementwise bitwise NOT: !a.

unsafe fn min_reduce(v: Self::Vector) -> T

Horizontal minimum across all lanes.

unsafe fn max_reduce(v: Self::Vector) -> T

Horizontal maximum across all lanes.

unsafe fn popcount(a: Self::Vector) -> Self::Vector

Elementwise population count (number of set bits).

unsafe fn horizontal_bitwise_and(v: Self::Vector) -> T

Horizontal bitwise AND across all lanes.

unsafe fn horizontal_bitwise_or(v: Self::Vector) -> T

Horizontal bitwise OR across all lanes.

unsafe fn horizontal_bitwise_xor(v: Self::Vector) -> T

Horizontal bitwise XOR across all lanes.

unsafe fn swap_adjacent(v: Self::Vector) -> Self::Vector

Swap each adjacent lane pair: [a0, a1, a2, a3, ...] -> [a1, a0, a3, a2, ...].

unsafe fn dup_even(v: Self::Vector) -> Self::Vector

Duplicate even lanes into odd lanes: [a0, a1, a2, a3, ...] -> [a0, a0, a2, a2, ...].

unsafe fn dup_odd(v: Self::Vector) -> Self::Vector

Duplicate odd lanes into even lanes: [a0, a1, a2, a3, ...] -> [a1, a1, a3, a3, ...].

unsafe fn fmaddsub( a: Self::Vector, b: Self::Vector, c: Self::Vector, ) -> Self::Vector

Alternating fused multiply: even lanes a*b - c, odd lanes a*b + c.

unsafe fn fmsubadd( a: Self::Vector, b: Self::Vector, c: Self::Vector, ) -> Self::Vector

Alternating fused multiply: even lanes a*b + c, odd lanes a*b - c.

impl StructuralPartialEq for SveArch