Enum LlmBackendDispatch 

pub enum LlmBackendDispatch {
    Cpu(CpuLlmBackend),
    Metal(MetalLlmBackend),
}

Variants

Cpu(CpuLlmBackend)

Metal(MetalLlmBackend)

Implementations

impl LlmBackendDispatch

pub fn is_cpu(&self) -> bool

Returns true when the backend is CPU-only (thread-safe for concurrent compute calls). Returns false for GPU backends (Metal/MLX) whose command buffers do not support concurrent encoding from multiple threads.

pub fn from_kind(kind: LlmBackendKind) -> Result<Self>

pub fn from_kind_with_model_bytes( kind: LlmBackendKind, model_bytes: u64, ) -> Result<Self>

Select a backend, optionally accounting for the model’s weight footprint.

model_bytes is the total weight size in bytes (0 = unknown). On Apple Silicon (unified memory), Metal is chosen for Auto when the model fits with a 1.5× KV-cache headroom factor; otherwise CPU is used to avoid swapping GPU memory which kills throughput.

Trait Implementations

impl Clone for LlmBackendDispatch

fn clone(&self) -> LlmBackendDispatch

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LlmBackendDispatch

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl LlmBackend for LlmBackendDispatch

fn name(&self) -> &'static str

fn linear_3d(&self, x: &Tensor, weight: &Tensor) -> Result<Tensor>

fn rms_norm(&self, x: &Tensor, weight: &Tensor, eps: f32) -> Result<Tensor>

fn layer_norm( &self, x: &Tensor, weight: &Tensor, bias: Option<&Tensor>, eps: f32, ) -> Result<Tensor>

fn silu(&self, x: &Tensor) -> Result<Tensor>

fn gelu(&self, x: &Tensor) -> Result<Tensor>

fn add(&self, a: &Tensor, b: &Tensor) -> Result<Tensor>

fn mul(&self, a: &Tensor, b: &Tensor) -> Result<Tensor>

fn apply_rope_positions( &self, x: &Tensor, positions: &[usize], base: f32, ) -> Result<Tensor>

fn gqa_attention( &self, q: &Tensor, k: &Tensor, v: &Tensor, n_kv_heads: usize, causal: bool, ) -> Result<Tensor>

fn logits_from_hidden( &self, hidden: &Tensor, lm_head: &Tensor, ) -> Result<Vec<f32>>

fn all_logits_from_hidden( &self, hidden: &Tensor, lm_head: &Tensor, ) -> Result<Vec<Vec<f32>>>

Compute logits for ALL positions in the sequence. Returns seq_len vectors each of length vocab_size. Default impl delegates to the CPU reference kernel; backends may override.

fn linear_3d_bias( &self, x: &Tensor, weight: &Tensor, bias: Option<&Tensor>, ) -> Result<Tensor>

Linear projection with an optional bias added over the last dimension. Backend-agnostic: computes linear_3d then folds in the bias on the host, so every backend gets correct bias handling for free.

fn apply_rope_partial( &self, x: &Tensor, positions: &[usize], base: f32, rotary_dim: usize, ) -> Result<Tensor>

RoPE over only the first rotary_dim channels (Phi partial rotary). Computed on the CPU reference kernel for all backends — it is cheap and avoids backend-specific partial-rotary support.