Struct LLMHandle 

pub struct LLMHandle { /* private fields */ }

Handle to an LLM instance.

Implementations

impl LLMHandle

pub fn destroy(&self) -> Result<(), Box<dyn Error + Send + Sync>>

Destroys the LLM instance and releases its resources.

pub fn run( &self, rkllm_input: RKLLMInput, rkllm_infer_params: Option<RKLLMInferParam>, user_data: impl RkllmCallbackHandler + Send + Sync + 'static, ) -> Result<(), Box<dyn Error + Send + Sync>>

Runs an LLM inference task asynchronously.

Parameters

• rkllm_input: The input data for the LLM.
• rkllm_infer_params: Optional parameters for the inference task.
• user_data: The callback handler to process the results.

Returns

This function does not return a value directly. Instead, it starts an asynchronous operation and processes results via the provided callback handler.

pub fn load_prompt_cache( &self, cache_path: &str, ) -> Result<(), Box<dyn Error + Send + Sync>>

Loads a prompt cache from a file.

Parameters

• cache_path: The path to the prompt cache file.

pub fn release_prompt_cache(&self) -> Result<(), Box<dyn Error + Send + Sync>>

Release a prompt cache from a file.

pub fn abort(&self) -> Result<(), Box<dyn Error + Send + Sync>>

Aborts an ongoing LLM task

pub fn is_running(&self) -> Result<(), Box<dyn Error + Send + Sync>>

Checks if an LLM task is currently running.

pub fn load_lora( &self, lora_cfg: &RKLLMLoraAdapter, ) -> Result<(), Box<dyn Error + Send + Sync>>

Loads a Lora adapter into the LLM.

Trait Implementations

impl Clone for LLMHandle

fn clone(&self) -> LLMHandle

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LLMHandle

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Copy for LLMHandle

impl Send for LLMHandle

impl Sync for LLMHandle