Trait BackendDispatch 

pub trait BackendDispatch: BackendCaps {
    const MATVEC_FALLBACK: &'static str;

    // Required methods
    fn dense_enabled() -> bool;
    fn has_launch(kind: QuantKind) -> bool;
    fn launch_matvec(
        kind: QuantKind,
        weights: &[u8],
        x: &[f32],
        rows: usize,
        row_bytes: usize,
    ) -> Option<Result<Vec<f32>, BackendError>>;
}

What a backend can actually do, which needs its crate compiled in.

Required Associated Constants

const MATVEC_FALLBACK: &'static str

What [crate::weight_matrix::WeightMatrix::apply_gpu] will do next if this backend’s launch fails, named for the log line. A property of this backend’s position in [with_gpu_backends], not of the backend itself.

Required Methods

fn dense_enabled() -> bool

Whether dense matmuls should try this backend in this process. Decided once from the environment, then cached for the process lifetime. See env_or_probe for the grammar, which is shared.

fn has_launch(kind: QuantKind) -> bool

Whether a launch function exists for kind, asked without a device and without launching anything.

This is the question BackendCaps::matvec_kernel claims to answer, asked of the code that actually runs. They are two structures that must agree about one thing, and they cannot be merged: the kernel NAMES are needed on builds where the backend crate is not a dependency and these function pointers do not exist. So the agreement is a test — every_kind_a_compiled_backend_claims_can_actually_be_launched, over every backend and all 21 kinds — rather than the debug_assert! that used to guard it, which fired only for kinds a run actually reached and only in debug.

Q5_0 is why. It was in Metal’s capability table with no launch function behind it from the day it was added, so batched prefill ran on the GPU while single-token decode silently fell to the CPU, and a release build just ran slower.

fn launch_matvec( kind: QuantKind, weights: &[u8], x: &[f32], rows: usize, row_bytes: usize, ) -> Option<Result<Vec<f32>, BackendError>>

One matvec. None means “this backend has no kernel for kind”, which is a different answer from Some(Err(_)), “the kernel exists and the launch failed” — the caller logs only the second.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors