Skip to main content

sha3_kernel_hasher/
lib.rs

1//! # SHA3-Kernel-Hasher
2//!
3//! **Production-grade SHA3-512 (FIPS 202) in pure Rust with dual-mode
4//! compilation for userspace and Windows kernel drivers.**
5//!
6//! ## Overview
7//!
8//! This crate provides a self-contained implementation of the SHA3-512
9//! cryptographic hash function as standardised in FIPS PUB 202 (National
10//! Institute of Standards and Technology, 2015).  It includes the complete
11//! Keccak-f\[1600\] permutation, correct SHA3 domain separation and
12//! `pad10*1` padding, an incremental (streaming) API, and helper modules
13//! for hex encoding, constant-time comparison, and I/O-based hashing.
14//!
15//! ## Algorithm
16//!
17//! SHA3-512 is built on the *sponge construction* (Bertoni et al., 2011)
18//! operating on a 1600-bit state partitioned into a 576-bit rate (*r*)
19//! and a 1024-bit capacity (*c*).  The construction proceeds in two
20//! phases:
21//!
22//! 1. **Absorb** — The message is padded with domain byte `0x06` and the
23//!    `pad10*1` rule (FIPS 202, §6.1), split into 72-byte blocks, and
24//!    each block is XORed into the state followed by Keccak-f\[1600\].
25//! 2. **Squeeze** — The first 512 bits (64 bytes) of the post-absorb
26//!    state are extracted as the digest.
27//!
28//! The Keccak-f\[1600\] permutation applies 24 rounds, each consisting
29//! of five step mappings — θ (theta), ρ (rho), π (pi), χ (chi), and
30//! ι (iota) — to a 5×5 matrix of 64-bit lanes.
31//!
32//! ## Security Properties
33//!
34//! | Property | Security (bits) |
35//! |----------|----------------|
36//! | Collision resistance | 256 |
37//! | Preimage resistance | 512 |
38//! | Second preimage resistance | 512 |
39//!
40//! SHA3 is structurally distinct from the Merkle-Damgård family and is
41//! immune to length-extension attacks (Kelsey and Schneier, 2005).
42//!
43//! ## Modules
44//!
45//! | Module | Description |
46//! |--------|-------------|
47//! | [`hex`] | `Sha3Digest` wrapper with `Display`, `LowerHex`, `UpperHex`, hex encode/decode |
48//! | [`ct`] | Constant-time byte comparison (Bernstein, Lange and Schwabe, 2012) |
49//! | `io` | Streaming `std::io::Read` adapter and `hash_file` convenience (std only) |
50//! | [`kernel_safe`] | Processor state save/restore for kernel SIMD usage |
51//!
52//! ## Compilation Modes
53//!
54//! - **`std`** (default) — Full standard library, runtime SIMD detection,
55//!   `io` module enabled.
56//! - **`kernel`** — `no_std` + `alloc` for Windows kernel-mode drivers.
57//!   The `io` module is disabled; SIMD defaults to scalar.
58//!
59//! ## SIMD Status
60//!
61//! Runtime detection probes for AVX-512F/BW/DQ, AVX2+BMI2, and SSE4.2 via
62//! `is_x86_feature_detected!`, and is exposed through [`SimdFeatures`] and
63//! [`Sha3_512Kernel::simd_features`] so a caller can query what the current
64//! CPU supports.
65//!
66//! **AVX-512 single-message permutation (`std` + `x86_64` only):** real,
67//! not a stub. Setting [`PerformanceConfig::use_avx512`] to `true`
68//! dispatches every Keccak-f\[1600\] call in `hash()`/`update()`/
69//! `finalize()` to a vectorized implementation (`src/simd_avx512.rs`)
70//! when the CPU supports AVX-512F, cross-validated bit-for-bit against
71//! the scalar implementation (zero state, 2000 random states, 50 chained
72//! permutations, and the same XKCP known-answer values the scalar tests
73//! check). **It defaults to `false` because it is not a measured
74//! throughput win**: `cargo bench --bench sha3_bench` on this container's
75//! Xeon shows the AVX-512 path within noise of scalar at 64 KiB payloads,
76//! ~3-4% *slower* at 1 MiB, and ~10% slower at 4 KiB. A single
77//! Keccak-f\[1600\] instance has no independent lanes for one 512-bit
78//! register to exploit — π's cross-row mixing needs real shuffle/blend
79//! instructions (originally a memory-gather, which measured ~25-30%
80//! slower still; see `simd_avx512`'s module docs for that history) that
81//! scalar code simply doesn't pay for. This matches the wider Keccak-SIMD
82//! literature: single-instance vectorization is rarely profitable —
83//! real throughput gains come from hashing multiple independent messages
84//! in parallel, which is exactly what `Sha3_512Kernel::hash_many` does.
85//!
86//! **Multi-message batching (`hash_many`, `std` + `x86_64` only): a real
87//! win.** `Sha3_512Kernel::hash_many` hashes a batch of independent
88//! messages using the SIMD-parallel permutations in `src/simd_parallel.rs`
89//! — independent messages give the vector width actual parallel work,
90//! unlike single-message hashing above. Measured end to end
91//! (`cargo bench --bench sha3_bench sha3_512_hash_many`, 64 independent
92//! 4 KiB messages, this container's Xeon):
93//!
94//! | Path | Throughput | vs. scalar loop |
95//! |------|-----------|------------------|
96//! | scalar (one `hash()` per message) | ~182 MiB/s | 1.0x |
97//! | AVX2×4 batching | ~220 MiB/s | ~1.2x |
98//! | AVX-512×8 batching | ~1004 MiB/s | **~5.5x** |
99//!
100//! AVX-512×8 is a genuine, substantial win — 8 independent messages
101//! give an 8-lane register real parallel work with no gather/shuffle
102//! needed (ρ+π reduce to array indexing plus a uniform per-register
103//! rotate — see `simd_parallel`'s module docs for why this layout is so
104//! much more favorable than single-message vectorization). AVX2×4 is
105//! correct and a modest win, but far short of AVX-512: AVX2 has no
106//! 64-bit rotate instruction, so ρ is emulated as shift-left +
107//! shift-right + or, which eats most of the 4-way parallelism's benefit.
108//! Both are controlled by the same
109//! [`PerformanceConfig::use_avx512`]/[`PerformanceConfig::use_avx2`]
110//! flags as `hash()`; when neither applies, `hash_many` falls back to
111//! hashing each message individually via the scalar path.
112//!
113//! `prefetch_data`, `parallel_chunks`, and `chunk_size` remain no-ops.
114//!
115//! ## Correctness Validation
116//!
117//! - NIST CAVP SHA3-512 test vectors (empty, `0xCC`, `0x41FB`, `0xA3`×200)
118//! - XKCP Keccak-f\[1600\] known-answer tests (lanes 0–4)
119//! - Cross-validated against RustCrypto `sha3` v0.10 for 20 input lengths
120//! - Incremental vs. one-shot, determinism, avalanche, boundary conditions
121//!
122//! ## Quick Start
123//!
124//! ```rust
125//! use sha3_kernel_hasher::{Sha3_512Kernel, Sha3Digest};
126//!
127//! let mut hasher = Sha3_512Kernel::new();
128//!
129//! // One-shot with typed digest
130//! let digest: Sha3Digest = hasher.hash_digest(b"hello");
131//! println!("{}", digest);
132//!
133//! // Incremental hashing
134//! hasher.reset();
135//! hasher.update(b"hello");
136//! hasher.update(b" world");
137//! let digest = hasher.finalize_digest();
138//! println!("{:X}", digest);
139//! ```
140//!
141//! ## References
142//!
143//! - Bertoni, G. et al. (2011) *The Keccak Reference*, v3.0.
144//!   <https://keccak.team/files/Keccak-reference-3.0.pdf>
145//! - NIST (2015) *SHA-3 Standard*. FIPS PUB 202.
146//!   doi:10.6028/NIST.FIPS.202
147//! - Bernstein, D.J. (2005) *Cache-timing attacks on AES*.
148//!   <https://cr.yp.to/antiforgery/cachetiming-20050414.pdf>
149//! - Bernstein, D.J., Lange, T. and Schwabe, P. (2012)
150//!   'The Security Impact of a New Cryptographic Library',
151//!   *LATINCRYPT 2012*, LNCS 7533, pp. 159–176.
152//! - Kelsey, J. and Schneier, B. (2005) 'Second Preimages on n-Bit
153//!   Hash Functions for Much Less than 2^n Work', *EUROCRYPT 2005*,
154//!   LNCS 3494, pp. 474–490.
155//! - Kocher, P.C. (1996) 'Timing Attacks on Implementations of
156//!   Diffie-Hellman, RSA, DSS, and Other Systems', *CRYPTO '96*,
157//!   LNCS 1109, pp. 104–113.
158
159#![cfg_attr(feature = "kernel", no_std)]
160#![warn(missing_docs)]
161
162#[cfg(feature = "kernel")]
163extern crate alloc;
164
165#[cfg(feature = "kernel")]
166use alloc::vec::Vec;
167#[cfg(not(feature = "kernel"))]
168use std::vec::Vec;
169
170#[cfg(feature = "serde")]
171use serde::{Deserialize, Serialize};
172
173pub mod ct;
174pub mod hex;
175mod keccak;
176
177#[cfg(not(feature = "kernel"))]
178pub mod io;
179
180// AVX-512 requires `is_x86_feature_detected!`, which is a `std` macro (no
181// `core` equivalent) — so this is gated out of the `kernel` (`no_std`)
182// build, same as `io` above, and out of non-x86_64 targets entirely.
183#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
184mod simd_avx512;
185
186// Multi-message parallel permutation (AVX2x4 / AVX-512x8) backing
187// `hash_many`. Same cfg gate as `simd_avx512` — see its comment above.
188#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
189mod simd_parallel;
190
191pub use hex::Sha3Digest;
192// Only the non-AVX-512 `permute()` fallback below calls this directly; the
193// x86_64+std build's `permute()` reaches the scalar path indirectly via
194// `simd_avx512::keccak_f1600_dispatch` instead.
195#[cfg(any(not(target_arch = "x86_64"), feature = "kernel"))]
196use keccak::keccak_f1600;
197
198/// Test-only call counters proving which permutation path actually ran.
199///
200/// Digest *output* is identical whether the scalar or AVX-512 path runs
201/// (that's the whole correctness contract in `simd_avx512.rs`), so an
202/// output-only test can't tell "the config flag is wired up" apart from
203/// "the config flag is silently ignored and everything happens to still
204/// be correct scalar code." These counters close that gap.
205#[cfg(test)]
206static AVX512_PATH_CALLS: core::sync::atomic::AtomicUsize = core::sync::atomic::AtomicUsize::new(0);
207#[cfg(test)]
208static SCALAR_PATH_CALLS: core::sync::atomic::AtomicUsize = core::sync::atomic::AtomicUsize::new(0);
209
210/// Single call site for the Keccak-f\[1600\] permutation used by the
211/// sponge (`Sha3_512State::absorb`/`finalize`). Chooses AVX-512 when
212/// `use_avx512` is `true` *and* the running CPU actually supports it
213/// (checked again, cheaply, inside [`simd_avx512::keccak_f1600_dispatch`]
214/// via `is_x86_feature_detected!`); falls back to the scalar
215/// implementation otherwise. On non-x86_64 targets, or the `kernel`
216/// (`no_std`) build where the AVX-512 module is compiled out entirely
217/// (see its `mod` declaration above), `use_avx512` is accepted but has
218/// no effect — there is no vector path to dispatch to.
219#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
220#[inline]
221fn permute(state: &mut [u64; 25], use_avx512: bool) {
222    #[cfg(test)]
223    {
224        use core::sync::atomic::Ordering;
225        if use_avx512 && is_x86_feature_detected!("avx512f") {
226            AVX512_PATH_CALLS.fetch_add(1, Ordering::Relaxed);
227        } else {
228            SCALAR_PATH_CALLS.fetch_add(1, Ordering::Relaxed);
229        }
230    }
231    simd_avx512::keccak_f1600_dispatch(state, use_avx512);
232}
233
234#[cfg(any(not(target_arch = "x86_64"), feature = "kernel"))]
235#[inline]
236fn permute(state: &mut [u64; 25], _use_avx512: bool) {
237    #[cfg(test)]
238    {
239        use core::sync::atomic::Ordering;
240        SCALAR_PATH_CALLS.fetch_add(1, Ordering::Relaxed);
241    }
242    keccak_f1600(state);
243}
244
245/// SHA3-512 rate in bytes: r = (1600 − 2×512) / 8 = 72.
246const SHA3_512_RATE: usize = 72;
247
248/// SHA3 domain separation byte (FIPS 202 §6.1).
249const SHA3_DOMAIN_BYTE: u8 = 0x06;
250
251/// SHA3-512 kernel hasher with SIMD optimisation
252pub struct Sha3_512Kernel {
253    state: Sha3_512State,
254    /// SIMD feature detection
255    features: SimdFeatures,
256    /// Performance configuration
257    config: PerformanceConfig,
258}
259
260/// SIMD feature detection results, as probed once via
261/// `is_x86_feature_detected!` and returned by
262/// [`Sha3_512Kernel::simd_features`].
263#[derive(Debug, Clone, Copy)]
264pub struct SimdFeatures {
265    /// Whether the running CPU supports AVX-512F/BW/DQ. Gates both the
266    /// single-message AVX-512 permutation and the AVX-512×8 batch path.
267    pub avx512_available: bool,
268    /// Whether the running CPU supports AVX2 + BMI2. Gates the AVX2×4
269    /// batch path used by `Sha3_512Kernel::hash_many` (`std` +
270    /// `x86_64` builds only — not an intra-doc link here since the
271    /// method doesn't exist under `kernel`/non-x86_64 builds).
272    pub avx2_available: bool,
273    /// Whether the running CPU supports SSE4.2. Currently detected but
274    /// not dispatched to by any permutation path.
275    pub sse42_available: bool,
276    /// Whether the running CPU supports BMI2 (`RORX`/`SHRX` etc.),
277    /// required alongside AVX2 for the AVX2×4 batch path.
278    pub bmi2_available: bool,
279}
280
281/// Performance configuration for hasher.
282///
283/// **`use_avx512`/`use_avx2` are real and wired up** on `std` + `x86_64`
284/// builds, but they mean different things depending on which method you
285/// call:
286///
287/// - **`Sha3_512Kernel::hash`/`update`/`finalize`** (single message):
288///   dispatch every Keccak-f\[1600\] permutation to
289///   [`simd_avx512`](crate) when `use_avx512` is `true` and the CPU
290///   supports AVX-512F (falling back to scalar otherwise; `use_avx2` has
291///   no effect here — see below). This is **not a clear win, which is
292///   why `use_avx512` defaults to `false`**: measured on this crate's own
293///   `cargo bench --bench sha3_bench` (this container's Xeon), it's
294///   within noise of scalar at 64 KiB payloads, ~3-4% *slower* at 1 MiB,
295///   and ~10% slower at 4 KiB. A single Keccak-f\[1600\] instance has no
296///   independent lanes for a 512-bit-wide register to parallelize
297///   *within* — π's cross-row mixing costs real shuffle/blend
298///   instructions that scalar code doesn't pay. Set it `true` here only
299///   if you've benchmarked your own workload and it actually helps.
300/// - **`Sha3_512Kernel::hash_many`** (a batch of independent messages):
301///   dispatches to AVX-512×8 or AVX2×4 batched permutations in
302///   `src/simd_parallel.rs`. Here the flags select a **real, measured
303///   win** — independent messages give the vector width genuine parallel
304///   work. See the crate-level "SIMD Status" section for full numbers;
305///   in short, AVX-512×8 measured ~5.5x over a scalar loop, AVX2×4 only
306///   ~1.2x (AVX2 has no 64-bit rotate instruction, so ρ is emulated via
307///   shift+shift+or, eating most of its 4-way parallelism).
308///
309/// `prefetch_data`, `parallel_chunks`, and `chunk_size` remain no-ops —
310/// nothing reads them yet.
311#[derive(Debug, Clone, Copy)]
312pub struct PerformanceConfig {
313    /// Real and wired up (`std` + `x86_64` only) for both `hash()` and
314    /// `hash_many()` — see the struct-level docs above; the two methods
315    /// get very different returns on setting this `true`.
316    pub use_avx512: bool,
317    /// No-op for `hash()` (AVX2 cannot accelerate a single-message
318    /// permutation — see struct docs); real, and a modest win, for
319    /// `hash_many()`.
320    pub use_avx2: bool,
321    /// No-op today; reserved for a future prefetch optimization.
322    pub prefetch_data: bool,
323    /// No-op today; reserved for a future parallel-chunk optimization.
324    pub parallel_chunks: bool,
325    /// No-op today.
326    pub chunk_size: usize,
327}
328
329impl Default for PerformanceConfig {
330    fn default() -> Self {
331        Self {
332            // Not a measured win on this crate's own benchmarks (see the
333            // struct docs) — opt-in, not default-on.
334            use_avx512: false,
335            use_avx2: true,
336            prefetch_data: true,
337            parallel_chunks: true,
338            chunk_size: 1024,
339        }
340    }
341}
342
343/// SHA3-512 sponge state (64-byte aligned for SIMD compatibility)
344#[repr(C, align(64))]
345#[derive(Debug, Clone)]
346pub struct Sha3_512State {
347    /// Keccak state: 5×5 matrix of 64-bit lanes (1600 bits)
348    pub state: [u64; 25],
349    /// Current position in the absorb buffer (0..rate)
350    pub pos: usize,
351    /// Rate in bytes (72 for SHA3-512)
352    pub rate: usize,
353    /// Absorb buffer — accumulates input until a full rate-block is ready
354    pub buffer: [u8; 200],
355}
356
357#[cfg(feature = "serde")]
358mod state_serde {
359    use super::Sha3_512State;
360    use core::fmt;
361    use serde::de::{self, MapAccess, Visitor};
362    use serde::ser::SerializeStruct;
363    use serde::{Deserialize, Deserializer, Serialize, Serializer};
364
365    #[cfg(feature = "kernel")]
366    use alloc::vec::Vec;
367    #[cfg(not(feature = "kernel"))]
368    use std::vec::Vec;
369
370    impl Serialize for Sha3_512State {
371        fn serialize<S: Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
372            let mut s = serializer.serialize_struct("Sha3_512State", 4)?;
373            s.serialize_field("state", &self.state.as_slice())?;
374            s.serialize_field("pos", &self.pos)?;
375            s.serialize_field("rate", &self.rate)?;
376            s.serialize_field("buffer", &self.buffer.as_slice())?;
377            s.end()
378        }
379    }
380
381    impl<'de> Deserialize<'de> for Sha3_512State {
382        fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
383            #[derive(serde::Deserialize)]
384            #[serde(field_identifier, rename_all = "lowercase")]
385            enum Field {
386                State,
387                Pos,
388                Rate,
389                Buffer,
390            }
391
392            struct StateVisitor;
393
394            impl<'de> Visitor<'de> for StateVisitor {
395                type Value = Sha3_512State;
396
397                fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
398                    f.write_str("struct Sha3_512State")
399                }
400
401                fn visit_map<M: MapAccess<'de>>(
402                    self,
403                    mut map: M,
404                ) -> Result<Sha3_512State, M::Error> {
405                    let mut state_lanes: Option<Vec<u64>> = None;
406                    let mut pos: Option<usize> = None;
407                    let mut rate: Option<usize> = None;
408                    let mut buffer_vec: Option<Vec<u8>> = None;
409
410                    while let Some(key) = map.next_key::<Field>()? {
411                        match key {
412                            Field::State => {
413                                state_lanes = Some(map.next_value::<Vec<u64>>()?);
414                            }
415                            Field::Pos => {
416                                pos = Some(map.next_value::<usize>()?);
417                            }
418                            Field::Rate => {
419                                rate = Some(map.next_value::<usize>()?);
420                            }
421                            Field::Buffer => {
422                                buffer_vec = Some(map.next_value::<Vec<u8>>()?);
423                            }
424                        }
425                    }
426
427                    let sv = state_lanes.ok_or_else(|| de::Error::missing_field("state"))?;
428                    let pv = pos.ok_or_else(|| de::Error::missing_field("pos"))?;
429                    let rv = rate.ok_or_else(|| de::Error::missing_field("rate"))?;
430                    let bv = buffer_vec.ok_or_else(|| de::Error::missing_field("buffer"))?;
431
432                    let mut state = [0u64; 25];
433                    if sv.len() != 25 {
434                        return Err(de::Error::invalid_length(sv.len(), &"25"));
435                    }
436                    state.copy_from_slice(&sv);
437
438                    let mut buffer = [0u8; 200];
439                    if bv.len() != 200 {
440                        return Err(de::Error::invalid_length(bv.len(), &"200"));
441                    }
442                    buffer.copy_from_slice(&bv);
443
444                    Ok(Sha3_512State {
445                        state,
446                        pos: pv,
447                        rate: rv,
448                        buffer,
449                    })
450                }
451            }
452
453            deserializer.deserialize_struct(
454                "Sha3_512State",
455                &["state", "pos", "rate", "buffer"],
456                StateVisitor,
457            )
458        }
459    }
460}
461
462/// Hash result type — 64 bytes (512 bits)
463pub type Sha3_512Hash = [u8; 64];
464
465/// Memory region descriptor for kernel hashing.
466///
467/// Passed to [`Sha3_512Kernel::hash_memory_regions`], which hashes each
468/// region for which `is_readable` is `true` and `size` is a nonzero value
469/// not exceeding [`MAX_HASH_SIZE`], skipping the rest. The caller is
470/// responsible for the region actually being valid, mapped memory at
471/// `base_address` for `size` bytes — see [`kernel_safe`] and
472/// [`Sha3_512Kernel::hash_memory_region`] for the `unsafe` contract this
473/// descriptor feeds into.
474#[derive(Debug, Clone)]
475#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
476pub struct MemoryRegion {
477    /// Starting virtual address of the region.
478    pub base_address: u64,
479    /// Length of the region in bytes.
480    pub size: usize,
481    /// Whether the region is safe to read. Regions with `is_readable ==
482    /// false` are skipped by [`Sha3_512Kernel::hash_memory_regions`].
483    pub is_readable: bool,
484    /// Whether the region is writable. Informational only — not
485    /// consulted by the hashing path, which only reads.
486    pub is_writable: bool,
487    /// Whether the region is executable. Informational only — not
488    /// consulted by the hashing path, which only reads.
489    pub is_executable: bool,
490}
491
492/// Maximum safe hash size for kernel operations (256 MiB)
493pub const MAX_HASH_SIZE: usize = 256 * 1024 * 1024;
494
495/// Default chunk size for parallel processing
496pub const DEFAULT_CHUNK_SIZE: usize = 1024;
497
498impl Sha3_512Kernel {
499    /// Create a new SHA3-512 kernel hasher with default configuration.
500    pub fn new() -> Self {
501        Self::with_config(PerformanceConfig::default())
502    }
503
504    /// Create a hasher with a custom performance configuration.
505    ///
506    /// # Examples
507    ///
508    /// ```
509    /// use sha3_kernel_hasher::{PerformanceConfig, Sha3_512Kernel};
510    ///
511    /// let mut hasher = Sha3_512Kernel::with_config(PerformanceConfig {
512    ///     use_avx2: true,
513    ///     ..Default::default()
514    /// });
515    /// let digest = hasher.hash(b"hello");
516    /// assert_eq!(digest.len(), 64);
517    /// ```
518    pub fn with_config(config: PerformanceConfig) -> Self {
519        let features = Self::detect_simd_features();
520        Self {
521            state: Sha3_512State::new(),
522            features,
523            config,
524        }
525    }
526
527    /// Hash `data` in a single call, returning the 64-byte SHA3-512 digest.
528    ///
529    /// Runs the AVX-512-vectorized Keccak-f\[1600\] permutation when
530    /// `self.config.use_avx512` is `true` *and* the CPU actually supports
531    /// AVX-512F (checked at each permutation call); otherwise runs the
532    /// scalar path. `self.config.use_avx2`/`prefetch_data`/
533    /// `parallel_chunks`/`chunk_size` remain no-ops — see the crate-level
534    /// "SIMD Status" section for why AVX2 alone can't accelerate a
535    /// single-message permutation.
536    pub fn hash(&mut self, data: &[u8]) -> Sha3_512Hash {
537        self.state.reset();
538        self.state.absorb(data, self.config.use_avx512);
539        self.state.finalize(self.config.use_avx512)
540    }
541
542    /// Hash a raw memory region (kernel-compatible).
543    ///
544    /// # Safety
545    ///
546    /// The caller must guarantee that `[base_address, base_address + size)`
547    /// is a valid, readable memory region for the lifetime of this call.
548    /// Returns `[0u8; 64]` if `size` is zero or exceeds `MAX_HASH_SIZE`.
549    ///
550    /// # Examples
551    ///
552    /// ```
553    /// use sha3_kernel_hasher::Sha3_512Kernel;
554    ///
555    /// let buffer = vec![0xABu8; 4096];
556    /// let mut hasher = Sha3_512Kernel::new();
557    /// let digest = hasher.hash_memory_region(buffer.as_ptr() as u64, buffer.len());
558    /// assert_eq!(digest, hasher.hash(&buffer));
559    /// ```
560    pub fn hash_memory_region(&mut self, base_address: u64, size: usize) -> Sha3_512Hash {
561        if size == 0 || size > MAX_HASH_SIZE {
562            return [0u8; 64];
563        }
564        let memory_slice = unsafe { core::slice::from_raw_parts(base_address as *const u8, size) };
565        self.hash(memory_slice)
566    }
567
568    /// Hash multiple memory regions, returning one digest per readable region.
569    pub fn hash_memory_regions(&mut self, regions: &[MemoryRegion]) -> Vec<Sha3_512Hash> {
570        regions
571            .iter()
572            .filter(|r| r.is_readable && r.size > 0 && r.size <= MAX_HASH_SIZE)
573            .map(|region| self.hash_memory_region(region.base_address, region.size))
574            .collect()
575    }
576
577    /// Hash multiple independent messages, using SIMD-parallel batches
578    /// when `self.config` allows and the CPU supports it.
579    ///
580    /// Unlike `hash()`'s [`PerformanceConfig::use_avx512`] (opt-in, not a
581    /// measured win — see the crate-level "SIMD Status" section),
582    /// **AVX-512×8 batching here is a real, measured win**: independent
583    /// messages give the vector width actual parallel work, unlike a
584    /// single permutation. Measured on this container's Xeon (raw
585    /// permutation throughput, `simd_parallel::timing_smoke`): AVX-512×8
586    /// is ~5.3x faster than 8 sequential scalar permutations. AVX2×4 is
587    /// implemented and correct but measured only ~1.0x (no real gain) —
588    /// AVX2 has no 64-bit rotate instruction, so ρ is emulated as
589    /// shift-left + shift-right + or, and that overhead roughly cancels
590    /// out the 4-way parallelism. Both are controlled by the same
591    /// [`PerformanceConfig::use_avx512`]/[`PerformanceConfig::use_avx2`]
592    /// flags as `hash()`; when neither applies (disabled in config, not
593    /// supported by the CPU, `kernel` build, or non-x86_64 target),
594    /// messages are hashed individually via the scalar path.
595    ///
596    /// Messages are batched in groups of the selected width (8 or 4);
597    /// within a group, variable-length messages are supported directly
598    /// — each message's own FIPS 202 `pad10*1` padding determines its own
599    /// block count, and its digest is captured the instant its own last
600    /// block is permuted, even while other lanes in the same group still
601    /// have blocks left. See `hash_batch` below for the algorithm.
602    ///
603    /// Returns one digest per input message, in the same order.
604    ///
605    /// # Examples
606    ///
607    /// ```
608    /// # #[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
609    /// # {
610    /// use sha3_kernel_hasher::Sha3_512Kernel;
611    ///
612    /// let hasher = Sha3_512Kernel::new();
613    /// let messages: [&[u8]; 3] = [b"first", b"second", b"third"];
614    /// let digests = hasher.hash_many(&messages);
615    /// assert_eq!(digests.len(), 3);
616    /// # }
617    /// ```
618    #[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
619    pub fn hash_many(&self, messages: &[&[u8]]) -> Vec<Sha3_512Hash> {
620        let width = if self.config.use_avx512 && is_x86_feature_detected!("avx512f") {
621            8
622        } else if self.config.use_avx2 && is_x86_feature_detected!("avx2") {
623            4
624        } else {
625            1
626        };
627
628        let mut results = Vec::with_capacity(messages.len());
629        for chunk in messages.chunks(width.max(1)) {
630            match width {
631                8 => results.extend(hash_batch_x8(chunk)),
632                4 => results.extend(hash_batch_x4(chunk)),
633                _ => {
634                    for m in chunk {
635                        results.push(Sha3_512Kernel::new().hash(m));
636                    }
637                }
638            }
639        }
640        results
641    }
642
643    /// Detect available SIMD features at runtime (userspace only).
644    #[cfg(all(target_arch = "x86_64", not(feature = "kernel")))]
645    fn detect_simd_features() -> SimdFeatures {
646        SimdFeatures {
647            avx512_available: is_x86_feature_detected!("avx512f")
648                && is_x86_feature_detected!("avx512bw")
649                && is_x86_feature_detected!("avx512dq"),
650            avx2_available: is_x86_feature_detected!("avx2") && is_x86_feature_detected!("bmi2"),
651            sse42_available: is_x86_feature_detected!("sse4.2"),
652            bmi2_available: is_x86_feature_detected!("bmi2"),
653        }
654    }
655
656    /// In kernel mode, default to scalar (SIMD requires KeSaveFloatingPointState).
657    /// SIMD can be enabled via PerformanceConfig after verifying processor state.
658    #[cfg(any(not(target_arch = "x86_64"), feature = "kernel"))]
659    fn detect_simd_features() -> SimdFeatures {
660        SimdFeatures {
661            avx512_available: false,
662            avx2_available: false,
663            sse42_available: false,
664            bmi2_available: false,
665        }
666    }
667
668    /// Get the SIMD features detected on the current CPU at construction
669    /// time (see [`SimdFeatures`] for what each flag gates).
670    ///
671    /// # Examples
672    ///
673    /// ```
674    /// use sha3_kernel_hasher::Sha3_512Kernel;
675    ///
676    /// let hasher = Sha3_512Kernel::new();
677    /// let features = hasher.simd_features();
678    /// println!("AVX-512 available: {}", features.avx512_available);
679    /// ```
680    pub fn simd_features(&self) -> SimdFeatures {
681        self.features
682    }
683
684    /// Get the hasher's current [`PerformanceConfig`].
685    pub fn config(&self) -> PerformanceConfig {
686        self.config
687    }
688
689    /// Replace the hasher's [`PerformanceConfig`].
690    ///
691    /// # Examples
692    ///
693    /// ```
694    /// use sha3_kernel_hasher::{PerformanceConfig, Sha3_512Kernel};
695    ///
696    /// let mut hasher = Sha3_512Kernel::new();
697    /// hasher.update_config(PerformanceConfig {
698    ///     use_avx2: true,
699    ///     ..hasher.config()
700    /// });
701    /// ```
702    pub fn update_config(&mut self, config: PerformanceConfig) {
703        self.config = config;
704    }
705
706    /// Hash `data` and return a [`Sha3Digest`] wrapper with formatting traits.
707    ///
708    /// This is a convenience method equivalent to
709    /// `Sha3Digest::new(self.hash(data))`.
710    pub fn hash_digest(&mut self, data: &[u8]) -> Sha3Digest {
711        Sha3Digest::new(self.hash(data))
712    }
713
714    /// Finalize current hash (for incremental hashing).
715    ///
716    /// Applies SHA3 padding and extracts the digest. After calling this
717    /// method, the hasher must be `reset()` before reuse.
718    pub fn finalize(&mut self) -> Sha3_512Hash {
719        self.state.finalize(self.config.use_avx512)
720    }
721
722    /// Finalize and return a [`Sha3Digest`] wrapper with formatting traits.
723    pub fn finalize_digest(&mut self) -> Sha3Digest {
724        Sha3Digest::new(self.finalize())
725    }
726
727    /// Reset hasher state for a new message.
728    pub fn reset(&mut self) {
729        self.state.reset();
730    }
731
732    /// Update hasher with additional data (incremental / streaming).
733    ///
734    /// Can be called multiple times; the final digest is obtained
735    /// by calling `finalize()`.
736    pub fn update(&mut self, data: &[u8]) {
737        self.state.absorb(data, self.config.use_avx512);
738    }
739}
740
741impl Default for Sha3_512Kernel {
742    fn default() -> Self {
743        Self::new()
744    }
745}
746
747// ---------------------------------------------------------------------------
748// hash_many — SIMD-parallel batch hashing of independent messages.
749//
750// This operates on raw `[u64; 25]` states directly rather than going
751// through `Sha3_512State`, because a batch of W independent sponges needs
752// W state arrays live at once for the vector primitives in
753// `simd_parallel` to operate on — `Sha3_512State`'s single-sponge,
754// streaming (`pos`/`buffer`) design isn't the right shape for that, and
755// retrofitting it would complicate the streaming API this function
756// doesn't need. `pad_message`/`xor_block_into`/`squeeze` below
757// deliberately mirror `Sha3_512State::absorb`/`finalize`'s byte-level
758// logic exactly (same rate, same pad10*1 rule, same little-endian word
759// order) — `tests::hash_many_matches_hash_for_every_length_0_to_300`
760// cross-checks the two against each other so they can't silently drift.
761// ---------------------------------------------------------------------------
762
763#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
764fn pad_message(msg: &[u8]) -> Vec<u8> {
765    let rate = SHA3_512_RATE;
766    let full_blocks = msg.len() / rate;
767    let total_len = (full_blocks + 1) * rate;
768    let mut buf = vec![0u8; total_len];
769    buf[..msg.len()].copy_from_slice(msg);
770    buf[msg.len()] = SHA3_DOMAIN_BYTE;
771    let last = total_len - 1;
772    buf[last] |= 0x80;
773    buf
774}
775
776#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
777fn xor_block_into(state: &mut [u64; 25], block: &[u8]) {
778    let words = SHA3_512_RATE / 8;
779    for (i, lane) in state.iter_mut().enumerate().take(words) {
780        let off = i * 8;
781        let word = u64::from_le_bytes([
782            block[off],
783            block[off + 1],
784            block[off + 2],
785            block[off + 3],
786            block[off + 4],
787            block[off + 5],
788            block[off + 6],
789            block[off + 7],
790        ]);
791        *lane ^= word;
792    }
793}
794
795#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
796fn squeeze(state: &[u64; 25]) -> Sha3_512Hash {
797    let mut hash = [0u8; 64];
798    for i in 0..8 {
799        hash[i * 8..(i + 1) * 8].copy_from_slice(&state[i].to_le_bytes());
800    }
801    hash
802}
803
804/// Batch up to 8 independent messages through `simd_parallel`'s
805/// AVX-512×8 permutation. `chunk.len()` must be `<= 8` — lanes beyond
806/// `chunk.len()` stay all-zero and are never read.
807///
808/// # Algorithm
809///
810/// Each message is padded independently ([`pad_message`]), so messages
811/// of different lengths need different numbers of blocks. Blocks are
812/// processed in lockstep across all lanes — every lane that still has a
813/// block at `block_idx` gets it XORed in before the shared vector
814/// permutation call; a lane with fewer blocks simply contributes no XOR
815/// once its own blocks are exhausted (it still gets permuted along with
816/// the others — harmless, since nothing reads its state again after its
817/// digest is captured). The instant a lane reaches its own last block
818/// (`block_idx == num_blocks[i] - 1`), its digest is squeezed out
819/// immediately, before any further block-XOR/permute rounds run for
820/// lanes that aren't done yet — that ordering is what makes variable-
821/// length batching correct.
822#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
823fn hash_batch_x8(chunk: &[&[u8]]) -> Vec<Sha3_512Hash> {
824    debug_assert!(chunk.len() <= 8);
825    let padded: Vec<Vec<u8>> = chunk.iter().map(|m| pad_message(m)).collect();
826    let num_blocks: Vec<usize> = padded.iter().map(|p| p.len() / SHA3_512_RATE).collect();
827    let max_blocks = num_blocks.iter().copied().max().unwrap_or(0);
828
829    let mut states = [[0u64; 25]; 8];
830    let mut results = vec![[0u8; 64]; chunk.len()];
831
832    for block_idx in 0..max_blocks {
833        for i in 0..chunk.len() {
834            if block_idx < num_blocks[i] {
835                let start = block_idx * SHA3_512_RATE;
836                xor_block_into(&mut states[i], &padded[i][start..start + SHA3_512_RATE]);
837            }
838        }
839        // SAFETY: this function is only reached from `hash_many`, which
840        // checked `is_x86_feature_detected!("avx512f")` before selecting
841        // width 8.
842        unsafe { simd_parallel::keccak_f1600_x8_avx512(&mut states) };
843        for i in 0..chunk.len() {
844            if block_idx == num_blocks[i] - 1 {
845                results[i] = squeeze(&states[i]);
846            }
847        }
848    }
849    results
850}
851
852/// Same as [`hash_batch_x8`], batching up to 4 messages through
853/// `simd_parallel`'s AVX2×4 permutation.
854#[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
855fn hash_batch_x4(chunk: &[&[u8]]) -> Vec<Sha3_512Hash> {
856    debug_assert!(chunk.len() <= 4);
857    let padded: Vec<Vec<u8>> = chunk.iter().map(|m| pad_message(m)).collect();
858    let num_blocks: Vec<usize> = padded.iter().map(|p| p.len() / SHA3_512_RATE).collect();
859    let max_blocks = num_blocks.iter().copied().max().unwrap_or(0);
860
861    let mut states = [[0u64; 25]; 4];
862    let mut results = vec![[0u8; 64]; chunk.len()];
863
864    for block_idx in 0..max_blocks {
865        for i in 0..chunk.len() {
866            if block_idx < num_blocks[i] {
867                let start = block_idx * SHA3_512_RATE;
868                xor_block_into(&mut states[i], &padded[i][start..start + SHA3_512_RATE]);
869            }
870        }
871        // SAFETY: this function is only reached from `hash_many`, which
872        // checked `is_x86_feature_detected!("avx2")` before selecting
873        // width 4.
874        unsafe { simd_parallel::keccak_f1600_x4_avx2(&mut states) };
875        for i in 0..chunk.len() {
876            if block_idx == num_blocks[i] - 1 {
877                results[i] = squeeze(&states[i]);
878            }
879        }
880    }
881    results
882}
883
884// ---------------------------------------------------------------------------
885// Sha3_512State — core sponge implementation
886// ---------------------------------------------------------------------------
887
888impl Default for Sha3_512State {
889    fn default() -> Self {
890        Self {
891            state: [0u64; 25],
892            pos: 0,
893            rate: SHA3_512_RATE,
894            buffer: [0u8; 200],
895        }
896    }
897}
898
899impl Sha3_512State {
900    /// Create a new SHA3-512 sponge state (all-zero per FIPS 202 §4).
901    pub fn new() -> Self {
902        Self::default()
903    }
904
905    /// Reset the sponge to the initial all-zero state.
906    pub fn reset(&mut self) {
907        self.state = [0u64; 25];
908        self.pos = 0;
909        self.buffer = [0u8; 200];
910    }
911
912    /// Absorb arbitrary-length input data into the sponge.
913    ///
914    /// Input is buffered until a full rate-block (72 bytes) is accumulated,
915    /// then XORed into the state and followed by a Keccak-f\[1600\]
916    /// permutation.
917    fn absorb(&mut self, data: &[u8], use_avx512: bool) {
918        let mut offset = 0;
919        let len = data.len();
920
921        // If we have buffered data, fill the buffer first
922        if self.pos > 0 {
923            let remaining = self.rate - self.pos;
924            let to_copy = core::cmp::min(remaining, len);
925            self.buffer[self.pos..self.pos + to_copy].copy_from_slice(&data[..to_copy]);
926            self.pos += to_copy;
927            offset += to_copy;
928
929            if self.pos == self.rate {
930                self.xor_block_into_state();
931                permute(&mut self.state, use_avx512);
932                self.pos = 0;
933            }
934        }
935
936        // Process full rate-blocks directly from input (zero-copy)
937        while offset + self.rate <= len {
938            // XOR rate bytes from data directly into state lanes
939            let block = &data[offset..offset + self.rate];
940            let words = self.rate / 8; // 9 words for SHA3-512
941            for i in 0..words {
942                let off = i * 8;
943                let word = u64::from_le_bytes([
944                    block[off],
945                    block[off + 1],
946                    block[off + 2],
947                    block[off + 3],
948                    block[off + 4],
949                    block[off + 5],
950                    block[off + 6],
951                    block[off + 7],
952                ]);
953                self.state[i] ^= word;
954            }
955            permute(&mut self.state, use_avx512);
956            offset += self.rate;
957        }
958
959        // Buffer any remaining bytes
960        let tail = len - offset;
961        if tail > 0 {
962            self.buffer[..tail].copy_from_slice(&data[offset..]);
963            self.pos = tail;
964        }
965    }
966
967    /// Apply SHA3 padding, absorb the final block, and squeeze the digest.
968    ///
969    /// Implements the SHA3 padding rule from FIPS 202 §6.1:
970    ///   1. Append domain separation byte `0x06`
971    ///   2. Pad with zeros
972    ///   3. Set the high bit of the last byte in the rate block (`0x80`)
973    ///   4. XOR the padded block into state
974    ///   5. Apply Keccak-f\[1600\]
975    ///   6. Extract 64 bytes in little-endian lane order
976    fn finalize(&mut self, use_avx512: bool) -> Sha3_512Hash {
977        // --- SHA3 padding (FIPS 202 §6.1) ---
978        // Clear buffer beyond current position
979        for i in self.pos..self.rate {
980            self.buffer[i] = 0;
981        }
982        // Domain separation byte
983        self.buffer[self.pos] = SHA3_DOMAIN_BYTE;
984        // pad10*1: set high bit of the last byte in the rate block
985        self.buffer[self.rate - 1] |= 0x80;
986
987        // Absorb the padded final block
988        self.xor_block_into_state();
989        permute(&mut self.state, use_avx512);
990
991        // --- Squeeze phase ---
992        // Extract 64 bytes (8 lanes × 8 bytes) in little-endian order
993        let mut hash = [0u8; 64];
994        for i in 0..8 {
995            hash[i * 8..(i + 1) * 8].copy_from_slice(&self.state[i].to_le_bytes());
996        }
997        hash
998    }
999
1000    /// XOR the current rate-block from the buffer into the sponge state.
1001    ///
1002    /// Reads `rate / 8` little-endian 64-bit words from `self.buffer`
1003    /// and XORs them into the first `rate / 8` lanes of `self.state`.
1004    #[inline]
1005    fn xor_block_into_state(&mut self) {
1006        let words = self.rate / 8; // 9 for SHA3-512
1007        for i in 0..words {
1008            let off = i * 8;
1009            let word = u64::from_le_bytes([
1010                self.buffer[off],
1011                self.buffer[off + 1],
1012                self.buffer[off + 2],
1013                self.buffer[off + 3],
1014                self.buffer[off + 4],
1015                self.buffer[off + 5],
1016                self.buffer[off + 6],
1017                self.buffer[off + 7],
1018            ]);
1019            self.state[i] ^= word;
1020        }
1021    }
1022}
1023
1024/// Kernel-safe memory operations
1025pub mod kernel_safe {
1026
1027    /// Save processor state for SIMD operations
1028    ///
1029    /// # Safety
1030    ///
1031    /// This function is unsafe because it directly manipulates processor registers.
1032    /// The caller must ensure:
1033    /// - The processor supports the required SIMD instructions (AVX2/AVX-512)
1034    /// - No other code is currently using SIMD registers
1035    /// - The saved state is properly restored before returning to normal execution
1036    /// - The function is called only from kernel mode or with appropriate privileges
1037    #[cfg(target_arch = "x86_64")]
1038    pub unsafe fn save_processor_state() -> ProcessorState {
1039        ProcessorState::save()
1040    }
1041
1042    /// Restore processor state after SIMD operations
1043    ///
1044    /// # Safety
1045    ///
1046    /// This function is unsafe because it directly manipulates processor registers.
1047    /// The caller must ensure:
1048    /// - The state was previously saved by `save_processor_state()`
1049    /// - No SIMD registers have been modified since the save
1050    /// - The processor is in the same state as when the save occurred
1051    /// - The function is called only from kernel mode or with appropriate privileges
1052    #[cfg(target_arch = "x86_64")]
1053    pub unsafe fn restore_processor_state(state: ProcessorState) {
1054        state.restore();
1055    }
1056
1057    /// Processor state structure
1058    #[cfg(target_arch = "x86_64")]
1059    #[repr(C)]
1060    pub struct ProcessorState {
1061        // XMM/YMM/ZMM registers
1062        simd_registers: [u8; 512],
1063        // Control registers
1064        control_registers: [u64; 8],
1065    }
1066
1067    #[cfg(target_arch = "x86_64")]
1068    impl ProcessorState {
1069        /// Save current processor state to memory
1070        ///
1071        /// # Safety
1072        ///
1073        /// This function is unsafe because it directly accesses processor registers.
1074        /// The caller must ensure:
1075        /// - The processor supports XSAVE/XSAVEOPT instructions
1076        /// - No other code is currently executing SIMD operations
1077        /// - Sufficient memory is allocated for the state structure
1078        /// - The function is called only from kernel mode or with appropriate privileges
1079        pub unsafe fn save() -> Self {
1080            // In real implementation, use XSAVE/XSAVEOPT
1081            Self {
1082                simd_registers: [0u8; 512],
1083                control_registers: [0u64; 8],
1084            }
1085        }
1086
1087        /// Restore processor state from saved memory
1088        ///
1089        /// # Safety
1090        ///
1091        /// This function is unsafe because it directly modifies processor registers.
1092        /// The caller must ensure:
1093        /// - The state was previously saved by the `save()` method
1094        /// - No processor registers have been modified since the save
1095        /// - The processor is in the same state as when the save occurred
1096        /// - The function is called only from kernel mode or with appropriate privileges
1097        pub unsafe fn restore(self) {
1098            // In real implementation, use XRSTOR
1099        }
1100    }
1101
1102    #[cfg(not(target_arch = "x86_64"))]
1103    pub struct ProcessorState;
1104
1105    #[cfg(not(target_arch = "x86_64"))]
1106    pub unsafe fn save_processor_state() -> ProcessorState {
1107        ProcessorState
1108    }
1109
1110    #[cfg(not(target_arch = "x86_64"))]
1111    pub unsafe fn restore_processor_state(_state: ProcessorState) {
1112        // No-op for non-x86_64
1113    }
1114}
1115
1116/// Performance utilities (available in std mode only)
1117#[cfg(not(feature = "kernel"))]
1118pub mod performance {
1119    use super::*;
1120
1121    /// Benchmark hasher performance
1122    pub fn benchmark_hasher<F>(hasher_fn: F, data: &[u8]) -> BenchmarkResult
1123    where
1124        F: Fn(&[u8]) -> Sha3_512Hash,
1125    {
1126        let start = std::time::Instant::now();
1127        let hash = hasher_fn(data);
1128        let duration = start.elapsed();
1129
1130        BenchmarkResult {
1131            hash,
1132            duration,
1133            throughput: (data.len() as f64) / duration.as_secs_f64(),
1134        }
1135    }
1136
1137    /// Benchmark result
1138    #[derive(Debug)]
1139    pub struct BenchmarkResult {
1140        /// Computed hash
1141        pub hash: Sha3_512Hash,
1142        /// Time taken
1143        pub duration: std::time::Duration,
1144        /// Throughput in bytes/second
1145        pub throughput: f64,
1146    }
1147}
1148
1149#[cfg(test)]
1150mod tests {
1151    use super::*;
1152
1153    /// Helper: encode byte array to lowercase hex string.
1154    fn bytes_to_hex(bytes: &[u8]) -> String {
1155        bytes.iter().map(|b| format!("{:02x}", b)).collect()
1156    }
1157
1158    // ===================================================================
1159    // NIST CAVP SHA3-512 Test Vectors (FIPS 202)
1160    // Source: https://csrc.nist.gov/projects/cryptographic-algorithm-
1161    //         validation-program/secure-hashing#sha3vsha3vss
1162    // ===================================================================
1163
1164    /// NIST ShortMsg vector: empty message (Len = 0)
1165    #[test]
1166    fn test_nist_sha3_512_empty() {
1167        let expected = "a69f73cca23a9ac5c8b567dc185a756e\
1168                        97c982164fe25859e0d1dcc1475c80a6\
1169                        15b2123af1f5f94c11e3e9402c3ac558\
1170                        f500199d95b6d3e301758586281dcd26";
1171        let mut hasher = Sha3_512Kernel::new();
1172        let hash = hasher.hash(b"");
1173        assert_eq!(
1174            bytes_to_hex(&hash),
1175            expected,
1176            "SHA3-512('') must match NIST CAVP vector"
1177        );
1178    }
1179
1180    /// NIST ShortMsg vector: single byte 0xCC (Len = 8)
1181    #[test]
1182    fn test_nist_sha3_512_one_byte_cc() {
1183        let expected = "3939fcc8b57b63612542da31a834e5dc\
1184                        c36e2ee0f652ac72e02624fa2e5adeec\
1185                        c7dd6bb3580224b4d6138706fc6e8059\
1186                        7b528051230b00621cc2b22999eaa205";
1187        let mut hasher = Sha3_512Kernel::new();
1188        let hash = hasher.hash(&[0xCC]);
1189        assert_eq!(
1190            bytes_to_hex(&hash),
1191            expected,
1192            "SHA3-512(0xCC) must match NIST CAVP vector"
1193        );
1194    }
1195
1196    /// NIST ShortMsg vector: two bytes 0x41, 0xFB (Len = 16)
1197    #[test]
1198    fn test_nist_sha3_512_two_bytes() {
1199        let expected = "aa092865a40694d91754dbc767b5202c\
1200                        546e226877147a95cb8b4c8f8709fe8c\
1201                        d6905256b089da37896ea5ca19d2cd9a\
1202                        b94c7192fc39f7cd4d598975a3013c69";
1203        let mut hasher = Sha3_512Kernel::new();
1204        let hash = hasher.hash(&[0x41, 0xFB]);
1205        assert_eq!(
1206            bytes_to_hex(&hash),
1207            expected,
1208            "SHA3-512(0x41FB) must match NIST CAVP vector"
1209        );
1210    }
1211
1212    /// NIST LongMsg vector: 200 bytes of 0xA3 repeated
1213    /// This exercises multi-block absorb (200 > rate of 72).
1214    #[test]
1215    fn test_nist_sha3_512_200_bytes_a3() {
1216        let expected = "e76dfad22084a8b1467fcf2ffa58361b\
1217                        ec7628edf5f3fdc0e4805dc48caeeca8\
1218                        1b7c13c30adf52a3659584739a2df46b\
1219                        e589c51ca1a4a8416df6545a1ce8ba00";
1220        let input = vec![0xA3u8; 200];
1221        let mut hasher = Sha3_512Kernel::new();
1222        let hash = hasher.hash(&input);
1223        assert_eq!(
1224            bytes_to_hex(&hash),
1225            expected,
1226            "SHA3-512(0xA3 × 200) must match NIST CAVP vector"
1227        );
1228    }
1229
1230    // ===================================================================
1231    // Incremental (streaming) API tests
1232    // ===================================================================
1233
1234    /// Verify that incremental hashing produces the same result
1235    /// as single-call hashing.
1236    #[test]
1237    fn test_incremental_matches_oneshot() {
1238        let data = b"The quick brown fox jumps over the lazy dog";
1239        let mut oneshot = Sha3_512Kernel::new();
1240        let expected = oneshot.hash(data);
1241
1242        let mut incremental = Sha3_512Kernel::new();
1243        incremental.reset();
1244        incremental.update(&data[..10]);
1245        incremental.update(&data[10..30]);
1246        incremental.update(&data[30..]);
1247        let result = incremental.finalize();
1248
1249        assert_eq!(result, expected, "Incremental must match oneshot");
1250    }
1251
1252    /// Verify byte-at-a-time feeding produces correct digest.
1253    #[test]
1254    fn test_incremental_byte_at_a_time() {
1255        let data = b"abcdefghijklmnopqrstuvwxyz";
1256        let mut oneshot = Sha3_512Kernel::new();
1257        let expected = oneshot.hash(data);
1258
1259        let mut inc = Sha3_512Kernel::new();
1260        inc.reset();
1261        for &b in data.iter() {
1262            inc.update(&[b]);
1263        }
1264        let result = inc.finalize();
1265        assert_eq!(result, expected, "Byte-at-a-time must match oneshot");
1266    }
1267
1268    // ===================================================================
1269    // Determinism and edge cases
1270    // ===================================================================
1271
1272    /// Same input must always produce the same output.
1273    #[test]
1274    fn test_deterministic() {
1275        let mut hasher = Sha3_512Kernel::new();
1276        let data = b"Bunker SHA3-512 determinism test";
1277        let h1 = hasher.hash(data);
1278        let h2 = hasher.hash(data);
1279        assert_eq!(h1, h2, "Hash must be deterministic");
1280    }
1281
1282    /// Different inputs must produce different outputs.
1283    #[test]
1284    fn test_avalanche() {
1285        let mut hasher = Sha3_512Kernel::new();
1286        let h1 = hasher.hash(b"input_a");
1287        let h2 = hasher.hash(b"input_b");
1288        assert_ne!(h1, h2, "Different inputs must avalanche");
1289    }
1290
1291    /// Hash of the empty message must NOT be all zeros.
1292    #[test]
1293    fn test_empty_not_zero() {
1294        let mut hasher = Sha3_512Kernel::new();
1295        let hash = hasher.hash(b"");
1296        assert_ne!(hash, [0u8; 64], "SHA3-512('') must not be all-zero");
1297    }
1298
1299    /// Boundary: exactly one rate-block (72 bytes).
1300    #[test]
1301    fn test_exact_rate_block() {
1302        let data = vec![0x42u8; 72];
1303        let mut hasher = Sha3_512Kernel::new();
1304        let h1 = hasher.hash(&data);
1305        let h2 = hasher.hash(&data);
1306        assert_eq!(h1, h2);
1307        assert_ne!(h1, [0u8; 64]);
1308    }
1309
1310    /// Boundary: rate-block + 1 byte (73 bytes → 2 blocks).
1311    #[test]
1312    fn test_rate_plus_one() {
1313        let data_72 = vec![0x42u8; 72];
1314        let data_73 = vec![0x42u8; 73];
1315        let mut hasher = Sha3_512Kernel::new();
1316        let h72 = hasher.hash(&data_72);
1317        let h73 = hasher.hash(&data_73);
1318        assert_ne!(h72, h73, "72 vs 73 bytes must differ");
1319    }
1320
1321    /// Large payload: 1 MiB of zeros.
1322    #[test]
1323    fn test_large_payload() {
1324        let data = vec![0u8; 1024 * 1024];
1325        let mut hasher = Sha3_512Kernel::new();
1326        let h1 = hasher.hash(&data);
1327        let h2 = hasher.hash(&data);
1328        assert_eq!(h1, h2);
1329        assert_ne!(h1, [0u8; 64]);
1330    }
1331
1332    // ===================================================================
1333    // SIMD feature detection
1334    // ===================================================================
1335
1336    #[test]
1337    fn test_simd_detection() {
1338        let hasher = Sha3_512Kernel::new();
1339        let features = hasher.simd_features();
1340        #[cfg(target_arch = "x86_64")]
1341        {
1342            // On x86_64, at minimum SSE4.2 should be available on any
1343            // CPU manufactured after 2008.
1344            assert!(
1345                features.sse42_available || features.avx2_available || features.avx512_available,
1346                "At least one SIMD tier must be detected on x86_64"
1347            );
1348        }
1349    }
1350
1351    // ===================================================================
1352    // PerformanceConfig::use_avx512 must actually change the permutation
1353    // path `hash()` takes — not just be stored and ignored. Output alone
1354    // can't distinguish "wired up" from "silently ignored, still
1355    // correct" since both paths are correct by construction; these
1356    // tests assert on `AVX512_PATH_CALLS`/`SCALAR_PATH_CALLS`, which
1357    // `permute()` increments on every real permutation call.
1358    // ===================================================================
1359
1360    #[test]
1361    #[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
1362    fn hash_dispatches_to_avx512_path_when_configured_and_available() {
1363        if !is_x86_feature_detected!("avx512f") {
1364            eprintln!("skipping: no avx512f on this host");
1365            return;
1366        }
1367        AVX512_PATH_CALLS.store(0, core::sync::atomic::Ordering::Relaxed);
1368
1369        let mut hasher = Sha3_512Kernel::with_config(PerformanceConfig {
1370            use_avx512: true,
1371            ..PerformanceConfig::default()
1372        });
1373        let vector_digest = hasher.hash(b"the quick brown fox jumps over the lazy dog");
1374
1375        assert!(
1376            AVX512_PATH_CALLS.load(core::sync::atomic::Ordering::Relaxed) > 0,
1377            "hash() with use_avx512: true must actually invoke the AVX-512 \
1378             permutation path on hardware that supports it, not just store \
1379             the flag"
1380        );
1381
1382        // Correctness: must still match the scalar-forced hasher exactly.
1383        let mut scalar_hasher = Sha3_512Kernel::with_config(PerformanceConfig {
1384            use_avx512: false,
1385            ..PerformanceConfig::default()
1386        });
1387        let scalar_digest = scalar_hasher.hash(b"the quick brown fox jumps over the lazy dog");
1388        assert_eq!(
1389            vector_digest, scalar_digest,
1390            "AVX-512-dispatched hash() must match scalar-dispatched hash() bit for bit"
1391        );
1392    }
1393
1394    #[test]
1395    #[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
1396    fn hash_dispatches_to_scalar_path_when_use_avx512_is_false() {
1397        SCALAR_PATH_CALLS.store(0, core::sync::atomic::Ordering::Relaxed);
1398
1399        let mut hasher = Sha3_512Kernel::with_config(PerformanceConfig {
1400            use_avx512: false,
1401            ..PerformanceConfig::default()
1402        });
1403        hasher.hash(b"some data");
1404
1405        assert!(
1406            SCALAR_PATH_CALLS.load(core::sync::atomic::Ordering::Relaxed) > 0,
1407            "hash() with use_avx512: false must take the scalar path"
1408        );
1409    }
1410
1411    #[test]
1412    #[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
1413    fn incremental_update_also_uses_configured_dispatch() {
1414        // absorb() is called on every full rate-block during multi-block
1415        // incremental hashing, not just once in finalize() — verify the
1416        // dispatch flag reaches that call site too, using a payload that
1417        // spans multiple 72-byte rate blocks.
1418        if !is_x86_feature_detected!("avx512f") {
1419            eprintln!("skipping: no avx512f on this host");
1420            return;
1421        }
1422        AVX512_PATH_CALLS.store(0, core::sync::atomic::Ordering::Relaxed);
1423
1424        let mut hasher = Sha3_512Kernel::with_config(PerformanceConfig {
1425            use_avx512: true,
1426            ..PerformanceConfig::default()
1427        });
1428        let data = vec![0xABu8; 72 * 5 + 13]; // 5 full blocks + a partial tail
1429        hasher.update(&data[..72 * 3]);
1430        hasher.update(&data[72 * 3..]);
1431        let vector_digest = hasher.finalize_digest();
1432
1433        let mut scalar_hasher = Sha3_512Kernel::with_config(PerformanceConfig {
1434            use_avx512: false,
1435            ..PerformanceConfig::default()
1436        });
1437        let scalar_digest = scalar_hasher.hash_digest(&data);
1438
1439        assert!(
1440            AVX512_PATH_CALLS.load(core::sync::atomic::Ordering::Relaxed) >= 5,
1441            "expected at least 5 AVX-512 permutation calls (one per full \
1442             rate-block plus finalize), got {}",
1443            AVX512_PATH_CALLS.load(core::sync::atomic::Ordering::Relaxed)
1444        );
1445        assert_eq!(vector_digest, scalar_digest);
1446    }
1447
1448    // ===================================================================
1449    // Memory region hashing
1450    // ===================================================================
1451
1452    #[test]
1453    fn test_memory_hashing() {
1454        let mut hasher = Sha3_512Kernel::new();
1455        let data = vec![0u8; 1024];
1456        let h1 = hasher.hash(&data);
1457        let h2 = hasher.hash_memory_region(data.as_ptr() as u64, data.len());
1458        assert_eq!(h1, h2, "Memory region hash must match slice hash");
1459    }
1460
1461    #[test]
1462    fn test_memory_hashing_zero_size() {
1463        let mut hasher = Sha3_512Kernel::new();
1464        let hash = hasher.hash_memory_region(0x1000, 0);
1465        assert_eq!(hash, [0u8; 64], "Zero-size region must return zero hash");
1466    }
1467
1468    #[test]
1469    fn test_memory_hashing_oversized() {
1470        let mut hasher = Sha3_512Kernel::new();
1471        let hash = hasher.hash_memory_region(0x1000, MAX_HASH_SIZE + 1);
1472        assert_eq!(hash, [0u8; 64], "Oversized region must return zero hash");
1473    }
1474
1475    // ===================================================================
1476    // Cross-validation against the `sha3` reference crate
1477    // ===================================================================
1478
1479    /// Verify our implementation matches the RustCrypto `sha3` crate
1480    /// for a variety of input lengths, covering all absorb edge cases.
1481    #[test]
1482    fn test_cross_validate_vs_sha3_crate() {
1483        use sha3::{Digest, Sha3_512 as RefSha3_512};
1484        let test_lengths: &[usize] = &[
1485            0, 1, 2, 31, 32, 63, 64, 71, 72, 73, 100, 143, 144, 145, 200, 255, 256, 512, 1000, 4096,
1486        ];
1487        let mut hasher = Sha3_512Kernel::new();
1488
1489        for &len in test_lengths {
1490            let data: Vec<u8> = (0..len).map(|i| (i & 0xFF) as u8).collect();
1491            let our_hash = hasher.hash(&data);
1492
1493            let mut ref_hasher = RefSha3_512::new();
1494            ref_hasher.update(&data);
1495            let ref_hash = ref_hasher.finalize();
1496
1497            assert_eq!(
1498                &our_hash[..],
1499                &ref_hash[..],
1500                "Mismatch vs sha3 crate at len={}",
1501                len
1502            );
1503        }
1504    }
1505
1506    // ===================================================================
1507    // hash_many — SIMD-parallel batch hashing
1508    // ===================================================================
1509
1510    #[cfg(all(not(feature = "kernel"), target_arch = "x86_64"))]
1511    mod hash_many_tests {
1512        use super::*;
1513
1514        fn force(use_avx512: bool, use_avx2: bool) -> Sha3_512Kernel {
1515            Sha3_512Kernel::with_config(PerformanceConfig {
1516                use_avx512,
1517                use_avx2,
1518                ..PerformanceConfig::default()
1519            })
1520        }
1521
1522        /// Every one of `hash_many`'s three dispatch paths (AVX-512x8,
1523        /// AVX2x4, scalar fallback) must match `hash()` called
1524        /// individually, across every rate-block boundary from 0 to
1525        /// just past 4 full blocks (0..=300 bytes covers the empty
1526        /// message, the single-byte, every tail length 1..71, exact
1527        /// multiples of the 72-byte rate, and multi-block messages).
1528        #[test]
1529        fn hash_many_matches_hash_for_every_length_0_to_300() {
1530            let lengths: Vec<usize> = (0..=300).collect();
1531            let messages: Vec<Vec<u8>> = lengths
1532                .iter()
1533                .map(|&len| (0..len).map(|i| (i & 0xFF) as u8).collect())
1534                .collect();
1535            let refs: Vec<&[u8]> = messages.iter().map(|m| m.as_slice()).collect();
1536
1537            let mut scalar_hasher = Sha3_512Kernel::new();
1538            let expected: Vec<Sha3_512Hash> =
1539                messages.iter().map(|m| scalar_hasher.hash(m)).collect();
1540
1541            for (label, hasher) in [
1542                ("avx512", force(true, false)),
1543                ("avx2", force(false, true)),
1544                ("scalar", force(false, false)),
1545            ] {
1546                if label == "avx512" && !is_x86_feature_detected!("avx512f") {
1547                    eprintln!("skipping hash_many({label}): no avx512f on this host");
1548                    continue;
1549                }
1550                if label == "avx2" && !is_x86_feature_detected!("avx2") {
1551                    eprintln!("skipping hash_many({label}): no avx2 on this host");
1552                    continue;
1553                }
1554                let results = hasher.hash_many(&refs);
1555                assert_eq!(results.len(), messages.len());
1556                for (i, &len) in lengths.iter().enumerate() {
1557                    assert_eq!(
1558                        results[i], expected[i],
1559                        "hash_many({label}) mismatch at message length {len}"
1560                    );
1561                }
1562            }
1563        }
1564
1565        /// Deliberately mixes wildly different lengths in one batch (one
1566        /// message finishes in a single block, another needs several) to
1567        /// exercise the "extract on completion, keep permuting other
1568        /// lanes" scheduling in `hash_batch_x8`/`hash_batch_x4` — this is
1569        /// the part that's wrong if a short message's digest gets
1570        /// overwritten by a later round meant for a longer sibling lane.
1571        #[test]
1572        fn hash_many_handles_mixed_lengths_in_one_batch() {
1573            let batch: Vec<Vec<u8>> = vec![
1574                vec![],                          // 1 block
1575                vec![0xAB; 5],                   // 1 block
1576                vec![0xCD; 72],                  // exact 1 rate-block of data -> 2 blocks total
1577                vec![0xEF; 72 * 3 + 17],         // multi-block, uneven tail
1578                b"the quick brown fox".to_vec(), // 1 block
1579            ];
1580            let refs: Vec<&[u8]> = batch.iter().map(|m| m.as_slice()).collect();
1581
1582            let mut scalar_hasher = Sha3_512Kernel::new();
1583            let expected: Vec<Sha3_512Hash> = batch.iter().map(|m| scalar_hasher.hash(m)).collect();
1584
1585            if is_x86_feature_detected!("avx512f") {
1586                let results = force(true, false).hash_many(&refs);
1587                assert_eq!(results, expected, "AVX-512x8 mixed-length batch mismatch");
1588            }
1589            if is_x86_feature_detected!("avx2") {
1590                let results = force(false, true).hash_many(&refs);
1591                assert_eq!(results, expected, "AVX2x4 mixed-length batch mismatch");
1592            }
1593        }
1594
1595        /// A batch larger than the SIMD width (8 or 4) must be chunked
1596        /// correctly, not just handle a single group.
1597        #[test]
1598        fn hash_many_handles_batches_larger_than_simd_width() {
1599            let batch: Vec<Vec<u8>> = (0..37)
1600                .map(|i| (0..(i * 3 + 1)).map(|b| (b & 0xFF) as u8).collect())
1601                .collect();
1602            let refs: Vec<&[u8]> = batch.iter().map(|m| m.as_slice()).collect();
1603
1604            let mut scalar_hasher = Sha3_512Kernel::new();
1605            let expected: Vec<Sha3_512Hash> = batch.iter().map(|m| scalar_hasher.hash(m)).collect();
1606
1607            let results = Sha3_512Kernel::new().hash_many(&refs);
1608            assert_eq!(results, expected);
1609        }
1610
1611        #[test]
1612        fn hash_many_empty_input_returns_empty() {
1613            let results = Sha3_512Kernel::new().hash_many(&[]);
1614            assert!(results.is_empty());
1615        }
1616    }
1617}
1618
1619/// Criterion benchmark harness for `cargo bench --features benchmarks`.
1620///
1621/// Not part of the crate's stable public API — it exists so
1622/// `cargo bench` can find a `main` entry point, and is only compiled
1623/// when the `benchmarks` feature is enabled. See the "Performance"
1624/// section of the crate-level docs for the throughput numbers this
1625/// harness produces.
1626///
1627/// `main` below is generated by the `criterion_main!` macro and can't
1628/// carry its own doc comment, hence the module-level `allow`.
1629#[cfg(feature = "benchmarks")]
1630#[allow(missing_docs)]
1631pub mod benchmarks {
1632    use super::*;
1633    #[cfg(feature = "kernel")]
1634    use alloc::vec;
1635    use criterion::{black_box, criterion_group, criterion_main, Criterion};
1636    #[cfg(not(feature = "kernel"))]
1637    use std::vec;
1638
1639    fn bench_sha3_512(c: &mut Criterion) {
1640        let mut hasher = Sha3_512Kernel::new();
1641        let data = vec![0u8; 1024 * 1024]; // 1MB
1642
1643        c.bench_function("sha3_512_hash", |b| {
1644            b.iter(|| black_box(hasher.hash(&data)))
1645        });
1646    }
1647
1648    criterion_group!(benches, bench_sha3_512);
1649    criterion_main!(benches);
1650}