Struct GenericBackend 

pub struct GenericBackend<D: Driver, A: ModelArch<D>> { /* private fields */ }

Generic backend that pairs a Driver with a ModelArch.

Implements EmbedBackend by calling arch.forward(driver, encodings). The driver provides hardware-specific compute primitives; the architecture orchestrates them into a full forward pass.

Lifetime invariant

_mmap must be declared after arch so it is dropped last. The architecture’s weight tensors reference pages in the memory-mapped file via zero-copy Metal buffers; dropping the mmap first would invalidate them.

Implementations

impl<D: Driver, A: ModelArch<D>> GenericBackend<D, A>

pub fn new( driver: D, arch: A, max_tokens: usize, is_gpu: bool, mmap: Mmap, ) -> Self

Create a new generic backend from a driver, architecture, and mmap.

The mmap must be the memory-mapped safetensors file whose pages back the weight tensors stored in arch. Create a new generic backend.

For GPU backends, runs a warm-up forward pass to prime the buffer pool. This is skipped for large models (max_tokens > 1024) where the warm-up cost exceeds the benefit. Create a new generic backend.

max_batch controls how many encodings are sent in each forward pass. Metal: 32 (optimal for M2 Max AMX). CUDA: 128+ (needs more work to saturate 128 SMs on RTX 4090).

pub fn with_max_batch( driver: D, arch: A, max_tokens: usize, is_gpu: bool, mmap: Mmap, max_batch: usize, ) -> Self

Create with explicit max batch size.

Trait Implementations

impl<D, A> EmbedBackend for GenericBackend<D, A>

where D: Driver + Send + Sync + 'static, A: ModelArch<D> + Send + Sync + 'static,

fn embed_batch(&self, encodings: &[Encoding]) -> Result<Vec<Vec<f32>>>

Embed a batch of pre-tokenized inputs, returning L2-normalized vectors.

fn supports_clone(&self) -> bool

Whether this backend supports cheap cloning for per-thread instances.

fn clone_backend(&self) -> Box<dyn EmbedBackend>

Create a cheap clone of this backend for per-thread use in rayon.

fn is_gpu(&self) -> bool

Whether this backend runs on a GPU.

fn max_tokens(&self) -> usize

Maximum token count this model supports (position embedding limit).