1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
//! Runtime-dispatched SIMD kernels (AVX2 / AVX-512, with a scalar fallback).
//!
//! Covers the two hot loops that dominate CPU time: the meet-solver's
//! candidate-vertex enumeration (`f64`) and the spectral raytracer's
//! plane-intersection and per-channel absorption math (`f32`).
//!
//! # Determinism contract
//!
//! Every kernel here is **bit-identical across dispatch levels** (scalar,
//! AVX2, AVX-512) and bit-identical to the scalar code it replaces, with one
//! documented exception:
//!
//! - The `f64` and `f32` geometry kernels replicate the exact operation order
//! of the `glam` scalar expressions they replace -- left-associated dot
//! products (`(x*x' + y*y') + z*z'`), `glam`'s cross/determinant/inverse
//! sequence, separate multiply and add (no FMA, matching `glam`'s scalar
//! arithmetic), and per-lane decisions replayed in ascending plane/triple
//! order. ([`exp_f32x8`] is the one FMA user: its scalar lane uses
//! `mul_add` and its vector body the fused intrinsics, so the two stay
//! bit-identical -- which is why the dispatch levels also require the `fma`
//! feature.) IEEE arithmetic is exactly rounded per operation, so identical
//! operation sequences give identical bits at any vector width.
//! - [`exp_f32x8`] is a polynomial exponential that is bit-identical across
//! its own dispatch levels but deliberately **not** identical to
//! `f32::exp` (libm): a vectorizable exponential cannot reproduce libm
//! bit-for-bit. Callers switching to it change results by a couple of ULP,
//! once, uniformly across machines -- a loud, re-baselined change, per this
//! crate's golden-test convention.
//!
//! No cross-lane floating-point reductions feed any decision: horizontal
//! steps only *select* existing lane values, with ties broken toward the
//! lowest plane index, matching the sequential scans they replace.
pub use exp_f32x8;
pub use ;
pub use ;
pub use ;
use OnceLock;
/// Which instruction set the kernels dispatch to on this machine.
static LEVEL: = new;
/// Detected dispatch level, cached for the process lifetime.
///
/// The environment variable `INDICATRIX_SIMD` (read once, at first call) can cap the
/// level below what the CPU supports: `scalar`, `avx2` or `avx512`. A cap never
/// raises the level above what was detected. Because every kernel is bit-identical
/// across levels (see the module docs), capping changes only speed, never output --
/// it exists so a profile-guided build (`scripts/pgo-build.ps1`) can train the scalar
/// kernels on an AVX-capable machine, and so a level can be A/B-timed in place.
const
// Two separate `cfg`-gated definitions, rather than one function with an internal
// `#[cfg]` block -- see `renderer::tonemap::effective_thread_count`'s identical split
// for the general pattern. `is_x86_feature_detected!` is a runtime CPUID check, not
// `const`-compatible, so the `x86_64` arm can never be `const fn`; every other
// architecture's arm is trivially `SimdLevel::Scalar` and clippy's
// `missing_const_for_fn` is right to ask for `const` there once the `x86_64`-only
// branches are gone.
const