Struct LLMVMCore

pub struct LLMVMCore { /* private fields */ }

The llmvm core which handles sending generation requests to the backend, managing message threads, model presets, and prompt templates. Frontends request text generation by invoking the core.

Implementations

impl LLMVMCore

pub async fn new(config: LLMVMCoreConfig) -> Result<Self>

pub async fn close_client(&self, backend_name: &str)

Close the active stdio (local child process) or HTTP client (remote process) for a given backend. If the backend client is over stdio, the backend child process will be killed.

Trait Implementations

impl Core for LLMVMCore

fn generate<'life0, 'async_trait>( &'life0 self, request: GenerationRequest, ) -> Pin<Box<dyn Future<Output = Result<GenerationResponse, ProtocolError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Generate text and return the whole response.

fn generate_stream<'life0, 'async_trait>( &'life0 self, request: GenerationRequest, ) -> Pin<Box<dyn Future<Output = Result<NotificationStream<GenerationResponse>, ProtocolError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Request text generation and return an asynchronous stream of generated tokens.

fn get_last_thread_info<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Option<ThreadInfo>, ProtocolError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Retrieve information for the last modified thread in the current project or user data directory.

fn get_all_thread_infos<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<Vec<ThreadInfo>, ProtocolError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Retrieve information for all available threads.

fn get_thread_messages<'life0, 'async_trait>( &'life0 self, id: String, ) -> Pin<Box<dyn Future<Output = Result<Vec<Message>, ProtocolError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Retrieve all thread messages for a thread id.

fn init_project(&self) -> Result<(), ProtocolError>

Initialize a new llmvm project in the current directory.