Enum ForwardEngine 

pub enum ForwardEngine {
    Llama(LlamaForward),
    Phi(PhiForward),
}

Architecture-specific inference engine with KV-cache support.

Variants

Llama(LlamaForward)

Phi(PhiForward)

Implementations

impl ForwardEngine

pub fn from_pretrained( info: ModelInfo, weight_paths: &[PathBuf], ) -> Result<Self>

pub fn from_weight_paths( info: ModelInfo, weight_paths: &[PathBuf], ) -> Result<Self>

pub fn from_weight_paths_with_backend( info: ModelInfo, weight_paths: &[PathBuf], backend: LlmBackendKind, ) -> Result<Self>

pub fn from_gguf(info: ModelInfo, path: &Path) -> Result<Self>

pub fn from_gguf_with_backend( info: ModelInfo, path: &Path, backend: LlmBackendKind, ) -> Result<Self>

pub fn from_gguf_mmap_with_backend( info: ModelInfo, path: &Path, backend: LlmBackendKind, ) -> Result<Self>

Load via memory-mapping — Q4_0/Q8_0 tensors are zero-copy from disk. The OS pages in weight blocks on demand; only active layers are resident.

pub fn reset_cache(&mut self)

pub fn forward_logits( &mut self, input_ids: &[u32], use_cache: bool, ) -> Result<Vec<f32>>

pub fn forward_all_logits(&mut self, input_ids: &[u32]) -> Result<Vec<Vec<f32>>>

Run the model on input_ids WITHOUT updating the KV cache and return logits for ALL positions. Used by speculative decoding to verify K draft tokens in a single target-model forward pass.

pub fn embed(&mut self, input_ids: &[u32]) -> Result<Vec<f32>>