Struct InferenceEngine 

pub struct InferenceEngine { /* private fields */ }

Inference engine for accelerated forward passes.

Implementations

impl InferenceEngine

pub fn new(config: InferenceConfig) -> Result<InferenceEngine, InferenceError>

Create a new inference engine.

pub fn device(&self) -> &Device

Get the active device.

pub fn is_gpu(&self) -> bool

Check if running on GPU.

pub fn linear( &self, input: &Tensor, weight: &Tensor, bias: Option<&Tensor>, ) -> Result<Tensor, InferenceError>

Linear layer forward pass with optional ternary quantization.

Computes: output = input @ weight.T + bias

pub fn ternary_linear( &self, input: &Tensor, weight: &Tensor, bias: Option<&Tensor>, ) -> Result<Tensor, InferenceError>

Ternary linear layer (quantized weights).

Quantizes weights to ternary for memory-efficient inference.

pub fn batched_forward<F>( &self, inputs: &Tensor, forward_fn: F, ) -> Result<Tensor, InferenceError>

where F: Fn(&Tensor) -> Result<Tensor, InferenceError>,

Batched inference with automatic chunking.

Splits large batches into smaller chunks to fit in memory.

pub fn softmax( &self, input: &Tensor, dim: usize, ) -> Result<Tensor, InferenceError>

Apply softmax along specified dimension.

pub fn layer_norm( &self, input: &Tensor, weight: &Tensor, bias: &Tensor, eps: f64, ) -> Result<Tensor, InferenceError>

Apply layer normalization.

Trait Implementations

impl Debug for InferenceEngine

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.