Enum Int8Tier 

pub enum Int8Tier {
    Scalar,
    Autovec,
    NeonSdot,
    WasmSimd128,
}

An executable int8 dot-product route.

Every variant is exactly equal in i32 to Scalar on every input. NeonSdot exists only on aarch64 builds with the neon-dotprod feature and is dispatchable only where the CPU reports FEAT_DotProd at runtime.

Variants

Scalar

Portable left-to-right checked-free scalar loop; the reference every tier must equal.

Autovec

Portable eight-lane loop, retained ONLY as an A/B datapoint: measured ~15x SLOWER than Scalar at m=1 on M4 Pro (NE-001) — the manual lane structure defeats LLVM’s autovectorizer, while the plain Scalar shape vectorizes to memory bandwidth. Never the dispatch default.

NeonSdot

Hand SDOT island (aarch64 + FEAT_DotProd), four 16-byte accumulator streams.

WasmSimd128

Hand SIMD128 island (wasm32 + simd128), four 16-byte accumulator streams.

The browser’s equivalent of Self::NeonSdot. Unlike aarch64, wasm has no int8 dot for the autovectorizer to find, so without this tier a browser runs the byte-at-a-time Scalar loop.

Implementations

impl Int8Tier

pub const fn as_str(self) -> &'static str

Stable machine-readable route name.

Examples found in repository

examples/int8_shape_bench.rs (line 65)

fn main() {
    let tiers: Vec<Int8Tier> = Int8Tier::available();
    println!("int8 shape bench — tiers available: {:?}", {
        tiers.iter().map(|t| t.as_str()).collect::<Vec<_>>()
    });
    println!(
        "interleaved rounds={ROUNDS} (+{WARMUP_ROUNDS} warmup); per-sample = mean over calls in one round; cv% over rounds"
    );

    for &m in &[1_usize, 16] {
        println!(
            "\n== m = {m} {} ==",
            if m == 1 {
                "(decode GEMV)"
            } else {
                "(seq-16 verify GEMM)"
            }
        );
        for &(label, n, k) in SHAPES {
            let calls: usize = (32 / m).max(2);
            let weight = pseudo_random_f32(n * k, 0xbe0_0001 ^ (n as u64) << 20 ^ k as u64);
            let x = pseudo_random_f32(m * k, 0xbe0_0002 ^ (m as u64) << 32 ^ k as u64);
            let quantized = QuantizedMatrix::quantize(&weight, n, k);
            let mut out = vec![0.0_f32; m * n];
            let mut x_q = vec![0_i8; m * k];
            let mut x_scales = vec![0.0_f32; m];

            // One arm per route, all interleaved inside every round.
            let mut f32_samples = Vec::with_capacity(ROUNDS);
            let mut tier_samples: Vec<Vec<f64>> =
                tiers.iter().map(|_| Vec::with_capacity(ROUNDS)).collect();

            for round in 0..ROUNDS + WARMUP_ROUNDS {
                // f32 arm
                let start = Instant::now();
                for _ in 0..calls {
                    f32ref::linear(
                        black_box(&x),
                        black_box(&weight),
                        None,
                        m,
                        k,
                        n,
                        black_box(&mut out),
                    );
                }
                let f32_us = start.elapsed().as_secs_f64() * 1e6 / calls as f64;

                // W8A8 arms, including dynamic activation quantization each call.
                let mut this_round = Vec::with_capacity(tiers.len());
                for &tier in &tiers {
                    let start = Instant::now();
                    for _ in 0..calls {
                        for ((x_row, q_row), scale) in x
                            .chunks_exact(k)
                            .zip(x_q.chunks_exact_mut(k))
                            .zip(x_scales.iter_mut())
                        {
                            *scale = quantize_row_q8(black_box(x_row), q_row);
                        }
                        linear_q8(
                            black_box(&x_q),
                            black_box(&x_scales),
                            black_box(&quantized),
                            None,
                            m,
                            black_box(&mut out),
                            tier,
                        );
                    }
                    this_round.push(start.elapsed().as_secs_f64() * 1e6 / calls as f64);
                }

                if round >= WARMUP_ROUNDS {
                    f32_samples.push(f32_us);
                    for (samples, sample) in tier_samples.iter_mut().zip(&this_round) {
                        samples.push(*sample);
                    }
                }
            }

            let f32_stats = stats(&f32_samples);
            // The f32 reference loops rows outermost, so it streams the weight matrix once per
            // activation row (m times per call); the q8 kernel is weight-stationary and streams
            // it exactly once per call. The column reports actual weight bytes moved per second.
            let f32_bytes = (n * k * 4 * m) as f64;
            println!(
                "{label}  f32     {:9.1} us  cv {:4.1}%  ({:5.1} GB/s weight-stream)",
                f32_stats.mean_us,
                f32_stats.cv_percent,
                f32_bytes / (f32_stats.mean_us * 1e-6) / 1e9,
            );
            for (tier, samples) in tiers.iter().zip(&tier_samples) {
                let tier_stats = stats(samples);
                let q8_bytes = (n * k) as f64;
                let verdict = if tier_stats.cv_percent > 5.0 || f32_stats.cv_percent > 5.0 {
                    "REFUSED (cv>5%)"
                } else {
                    ""
                };
                println!(
                    "{label}  q8 {:9} {:9.1} us  cv {:4.1}%  ({:5.1} GB/s weight-stream)  x{:.2} vs f32 {verdict}",
                    tier.as_str(),
                    tier_stats.mean_us,
                    tier_stats.cv_percent,
                    q8_bytes / (tier_stats.mean_us * 1e-6) / 1e9,
                    f32_stats.mean_us / tier_stats.mean_us,
                );
            }
        }
    }
    println!(
        "\nNOTE: ratios above compare routes inside this tree (self-comparison = maintenance),\nnever a pinned incumbent. cv%>5 rows are refused, not averaged."
    );
}

pub fn available() -> Vec<Self>

Every tier this build can execute on the running silicon, scalar first.

Examples found in repository

examples/int8_shape_bench.rs (line 63)

fn main() {
    let tiers: Vec<Int8Tier> = Int8Tier::available();
    println!("int8 shape bench — tiers available: {:?}", {
        tiers.iter().map(|t| t.as_str()).collect::<Vec<_>>()
    });
    println!(
        "interleaved rounds={ROUNDS} (+{WARMUP_ROUNDS} warmup); per-sample = mean over calls in one round; cv% over rounds"
    );

    for &m in &[1_usize, 16] {
        println!(
            "\n== m = {m} {} ==",
            if m == 1 {
                "(decode GEMV)"
            } else {
                "(seq-16 verify GEMM)"
            }
        );
        for &(label, n, k) in SHAPES {
            let calls: usize = (32 / m).max(2);
            let weight = pseudo_random_f32(n * k, 0xbe0_0001 ^ (n as u64) << 20 ^ k as u64);
            let x = pseudo_random_f32(m * k, 0xbe0_0002 ^ (m as u64) << 32 ^ k as u64);
            let quantized = QuantizedMatrix::quantize(&weight, n, k);
            let mut out = vec![0.0_f32; m * n];
            let mut x_q = vec![0_i8; m * k];
            let mut x_scales = vec![0.0_f32; m];

            // One arm per route, all interleaved inside every round.
            let mut f32_samples = Vec::with_capacity(ROUNDS);
            let mut tier_samples: Vec<Vec<f64>> =
                tiers.iter().map(|_| Vec::with_capacity(ROUNDS)).collect();

            for round in 0..ROUNDS + WARMUP_ROUNDS {
                // f32 arm
                let start = Instant::now();
                for _ in 0..calls {
                    f32ref::linear(
                        black_box(&x),
                        black_box(&weight),
                        None,
                        m,
                        k,
                        n,
                        black_box(&mut out),
                    );
                }
                let f32_us = start.elapsed().as_secs_f64() * 1e6 / calls as f64;

                // W8A8 arms, including dynamic activation quantization each call.
                let mut this_round = Vec::with_capacity(tiers.len());
                for &tier in &tiers {
                    let start = Instant::now();
                    for _ in 0..calls {
                        for ((x_row, q_row), scale) in x
                            .chunks_exact(k)
                            .zip(x_q.chunks_exact_mut(k))
                            .zip(x_scales.iter_mut())
                        {
                            *scale = quantize_row_q8(black_box(x_row), q_row);
                        }
                        linear_q8(
                            black_box(&x_q),
                            black_box(&x_scales),
                            black_box(&quantized),
                            None,
                            m,
                            black_box(&mut out),
                            tier,
                        );
                    }
                    this_round.push(start.elapsed().as_secs_f64() * 1e6 / calls as f64);
                }

                if round >= WARMUP_ROUNDS {
                    f32_samples.push(f32_us);
                    for (samples, sample) in tier_samples.iter_mut().zip(&this_round) {
                        samples.push(*sample);
                    }
                }
            }

            let f32_stats = stats(&f32_samples);
            // The f32 reference loops rows outermost, so it streams the weight matrix once per
            // activation row (m times per call); the q8 kernel is weight-stationary and streams
            // it exactly once per call. The column reports actual weight bytes moved per second.
            let f32_bytes = (n * k * 4 * m) as f64;
            println!(
                "{label}  f32     {:9.1} us  cv {:4.1}%  ({:5.1} GB/s weight-stream)",
                f32_stats.mean_us,
                f32_stats.cv_percent,
                f32_bytes / (f32_stats.mean_us * 1e-6) / 1e9,
            );
            for (tier, samples) in tiers.iter().zip(&tier_samples) {
                let tier_stats = stats(samples);
                let q8_bytes = (n * k) as f64;
                let verdict = if tier_stats.cv_percent > 5.0 || f32_stats.cv_percent > 5.0 {
                    "REFUSED (cv>5%)"
                } else {
                    ""
                };
                println!(
                    "{label}  q8 {:9} {:9.1} us  cv {:4.1}%  ({:5.1} GB/s weight-stream)  x{:.2} vs f32 {verdict}",
                    tier.as_str(),
                    tier_stats.mean_us,
                    tier_stats.cv_percent,
                    q8_bytes / (tier_stats.mean_us * 1e-6) / 1e9,
                    f32_stats.mean_us / tier_stats.mean_us,
                );
            }
        }
    }
    println!(
        "\nNOTE: ratios above compare routes inside this tree (self-comparison = maintenance),\nnever a pinned incumbent. cv%>5 rows are refused, not averaged."
    );
}

pub fn dispatch() -> Self

The route the int8 path dispatches by default, honoring the FTTS_INT8_TIER override.

The override exists for interleaved A/B measurement (scalar / autovec / neon-sdot); an unavailable or unrecognized override falls back to the measured default rather than panicking mid-synthesis. Until a per-shape KernelPlan lands, the default is NeonSdot where FEAT_DotProd exists, else Scalar. Measured on M4 Pro (2026-08-08, shape bench, noisy shared host, indicative): plain Scalar autovectorizes to ~50 GB/s and ties SDOT at m=1 — NE-INH-003 reconfirmed — while the hand-shaped Autovec lane loop defeats the vectorizer and loses ~15x; it stays only as an A/B datapoint.

Trait Implementations

impl Clone for Int8Tier

fn clone(&self) -> Int8Tier

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Int8Tier

impl Debug for Int8Tier

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Int8Tier

impl PartialEq for Int8Tier

fn eq(&self, other: &Int8Tier) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Int8Tier