Struct KernelDispatcher 

pub struct KernelDispatcher { /* private fields */ }

Dispatches kernel calls to the best available implementation.

Uses scirs2_core::simd::detect::CpuFeatures for CPU feature detection, ensuring consistent SIMD dispatch across the COOLJAPAN ecosystem.

Implementations

impl KernelDispatcher

pub fn auto_detect() -> Self

Create a dispatcher that auto-detects the best available kernel tier.

Queries SciRS2-Core’s cached CpuFeatures to determine the optimal tier for the current CPU.

pub fn with_tier(tier: KernelTier) -> Self

Create a dispatcher with a specific tier (for testing/benchmarks).

pub fn tier(&self) -> KernelTier

Get the selected kernel tier.

Trait Implementations

impl Debug for KernelDispatcher

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Fp8Kernel for KernelDispatcher

fn dequant_fp8_e4m3( &self, blocks: &[BlockFP8E4M3], output: &mut [f32], ) -> KernelResult<()>

Dequantize FP8 E4M3FN blocks — tier-aware SIMD dispatch.

fn dequant_fp8_e5m2( &self, blocks: &[BlockFP8E5M2], output: &mut [f32], ) -> KernelResult<()>

Dequantize FP8 E5M2 blocks — tier-aware SIMD dispatch.

fn gemv_fp8_e4m3( &self, blocks: &[BlockFP8E4M3], input: &[f32], output: &mut [f32], n_rows: usize, k: usize, ) -> KernelResult<()>

FP8 E4M3FN GEMV — tier-aware SIMD dispatch with optional GPU acceleration.

Dispatch priority on the KernelTier::Gpu path:

1. Metal (macOS + metal feature) — metal_gemv_fp8_e4m3.
2. CUDA (Linux/Windows + native-cuda feature) — cuda_gemv_fp8_e4m3.
3. CPU SIMD fallback (AVX-512 / AVX2 / NEON / scalar).

The raw-byte cast of blocks to *const u8 is sound because BlockFP8E4M3 is #[repr(C)] with size BLOCK_FP8_BYTES = 34.

fn gemv_fp8_e5m2( &self, blocks: &[BlockFP8E5M2], input: &[f32], output: &mut [f32], n_rows: usize, k: usize, ) -> KernelResult<()>

FP8 E5M2 GEMV — tier-aware SIMD dispatch with optional GPU acceleration.

Mirrors gemv_fp8_e4m3: Metal → CUDA → CPU SIMD. The raw-byte cast is sound because BlockFP8E5M2 is #[repr(C)] with size BLOCK_FP8_BYTES = 34.

fn gemm_fp8_e4m3( &self, blocks: &[BlockFP8E4M3], inputs: &[f32], outputs: &mut [f32], n_rows: usize, k: usize, batch: usize, ) -> KernelResult<()>

FP8 E4M3FN GEMM — tier-aware SIMD dispatch.

fn gemm_fp8_e5m2( &self, blocks: &[BlockFP8E5M2], inputs: &[f32], outputs: &mut [f32], n_rows: usize, k: usize, batch: usize, ) -> KernelResult<()>

FP8 E5M2 GEMM — tier-aware SIMD dispatch.

fn name_fp8(&self) -> &'static str

Display name for this FP8 kernel implementation.

impl OneBitKernel for KernelDispatcher

fn dequant( &self, blocks: &[BlockQ1_0G128], output: &mut [f32], ) -> KernelResult<()>

Dequantize blocks to FP32 values.

fn gemv( &self, blocks: &[BlockQ1_0G128], input: &[f32], output: &mut [f32], n_rows: usize, k: usize, ) -> KernelResult<()>

Fused 1-bit matrix × FP32 vector product (GEMV).

fn gemm( &self, blocks: &[BlockQ1_0G128], input: &[f32], output: &mut [f32], m: usize, n_rows: usize, k: usize, ) -> KernelResult<()>

Fused 1-bit matrix × FP32 matrix product (GEMM).

fn name(&self) -> &'static str

Display name for this kernel implementation.

fn is_gpu_accelerated(&self) -> bool

Whether this kernel routes ops through GPU hardware.

fn upload_weights(&self, blocks: &[BlockQ1_0G128]) -> Option<GpuWeightHandle>

Upload weight blocks to GPU memory for future cached GEMV/GEMM calls.

fn gemv_cached( &self, handle: GpuWeightHandle, input: &[f32], output: &mut [f32], n_rows: usize, k: usize, ) -> KernelResult<()>

GEMV using a pre-uploaded weight buffer (no host→device copy for weights).

fn batch_attn_phase( &self, hidden: &[f32], norm_weight: &[f32], norm_eps: f32, qkv_handle: GpuWeightHandle, q_rows: usize, k_rows: usize, h: usize, ) -> KernelResult<Option<(Vec<f32>, Vec<f32>, Vec<f32>)>>

Batch-accelerated attention input phase (RMSNorm + QKV in one command buffer).

fn batch_ffn_phase( &self, hidden: &mut [f32], attn_out: &[f32], norm_weight: &[f32], norm_eps: f32, attn_proj_handle: GpuWeightHandle, gate_up_handle: GpuWeightHandle, down_handle: GpuWeightHandle, h: usize, intermediate: usize, attn_proj_k: usize, ) -> KernelResult<bool>

Batch-accelerated FFN phase (attn_proj + residual + norm + gate_up + swiglu + down + residual).

impl TernaryKernel for KernelDispatcher

fn dequant_ternary_g128( &self, blocks: &[BlockTQ2_0_g128], output: &mut [f32], ) -> KernelResult<()>

Dequantize TQ2_0_g128 blocks to FP32 values.

fn gemv_ternary_g128( &self, blocks: &[BlockTQ2_0_g128], input: &[f32], output: &mut [f32], n_rows: usize, k: usize, ) -> KernelResult<()>

Fused ternary matrix × FP32 vector product (GEMV).

fn gemm_ternary_g128( &self, blocks: &[BlockTQ2_0_g128], input: &[f32], output: &mut [f32], m: usize, n_rows: usize, k: usize, ) -> KernelResult<()>

Fused ternary matrix × FP32 matrix product (GEMM).

fn upload_weights_ternary( &self, blocks: &[BlockTQ2_0_g128], ) -> Option<GpuWeightHandle>

Upload TQ2_0_g128 weight blocks to GPU memory for future cached GEMV calls.

fn gemv_ternary_g128_cached( &self, handle: GpuWeightHandle, input: &[f32], output: &mut [f32], n_rows: usize, k: usize, ) -> KernelResult<()>

GEMV using a pre-uploaded ternary weight buffer (no host→device copy for weights).