Struct Engine 

pub struct Engine { /* private fields */ }

A cloneable vllm.cpp serving engine.

Implementations

impl Engine

pub fn builder(model_path: impl Into<PathBuf>) -> EngineBuilder

Starts configuring an engine for a model directory or GGUF file.

pub fn load(model_path: impl Into<PathBuf>) -> Result<Self, Error>

Loads an engine using native defaults.

pub fn complete( &self, prompt: &str, params: &SamplingParams, ) -> Result<Completion, Error>

Runs one blocking text completion.

pub fn complete_stream<F>( &self, prompt: &str, params: &SamplingParams, callback: F, ) -> Result<StreamOutcome, Error>

where F: FnMut(StreamEvent) -> StreamControl,

Runs one blocking streaming text completion.

A callback panic is resumed only after native code has aborted the request and returned across the FFI boundary.

pub fn chat_json(&self, request_json: &str) -> Result<String, Error>

Runs one blocking OpenAI-style chat request and returns response JSON.

pub fn chat_stream_json<F>( &self, request_json: &str, callback: F, ) -> Result<StreamOutcome, Error>

where F: FnMut(StreamEvent) -> StreamControl,

Runs one blocking OpenAI-style streaming chat request.

pub fn chat(&self, request: &Value) -> Result<Value, Error>

impl Engine

pub fn submit<F>( &self, prompt: &str, params: &SamplingParams, callback: F, ) -> Result<Request, Error>

where F: FnMut(StreamEvent) -> StreamControl + Send + 'static,

Submits a non-blocking streaming completion to the shared engine.

The callback runs on a native delivery thread and receives an owned UTF-8 copy of each delta. A callback panic is contained and later reported by Request::wait as Error::CallbackPanicked.

Trait Implementations

impl Clone for Engine

fn clone(&self) -> Engine

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Engine

fn fmt(&self, formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.