Struct InferenceEngine 

pub struct InferenceEngine { /* private fields */ }

Implementations

impl InferenceEngine

pub fn new(api_key: Option<&str>, backend: InferenceBackend) -> Self

pub async fn chat_stream( &self, request: &InferenceRequest, on_text: &mut dyn FnMut(&str), ) -> Result<InferenceResponse>

Streaming chat — displays text tokens as they arrive, returns accumulated response

pub fn build_request( &self, messages: &[Message], system: Option<&str>, tools: &[ToolDefinition], model: Option<&str>, ) -> InferenceRequest

pub fn auto_route(&self, messages: &[Message]) -> &'static str

Auto-route: pick Opus for complex tasks, Haiku for simple ones. Returns the model ID to use based on the last user message.