Skip to main content

InferenceBackend

runtime::benchmark

Trait InferenceBackend 

Source 
pub trait InferenceBackend: Send + Sync {
    // Required methods
    fn name(&self) -> &str;
    fn load_model(&mut self, path: &Path) -> Result<f64>;
    fn generate(
        &mut self,
        prompt: &str,
        max_tokens: usize,
    ) -> Result<GenerationResult>;
    fn memory_usage(&self) -> u64;
    fn unload(&mut self);
}
Expand description

Trait for inference backends that can be benchmarked

Required Methods§

Source

fn name(&self) -> &str

Backend identifier name

Source

fn load_model(&mut self, path: &Path) -> Result<f64>

Load model from path, return load time in milliseconds

Source

fn generate( &mut self, prompt: &str, max_tokens: usize, ) -> Result<GenerationResult>

Generate text from prompt, return generation result

Source

fn memory_usage(&self) -> u64

Get current memory usage in bytes

Source

fn unload(&mut self)

Unload model and free resources

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§