Trait ChatModel

Source

pub trait ChatModel:
    Send
    + Sync
    + Clone
    + 'static {
    // Required methods
    fn invoke<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        messages: &'life1 [Message],
        options: Option<&'life2 CallOptions>,
    ) -> Pin<Box<dyn Future<Output = Result<Message, LlmError>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait;
    fn stream(
        &self,
        messages: &[Message],
        options: Option<&CallOptions>,
    ) -> BoxStream<'_, Result<MessageChunk, LlmError>>;
    fn bind_tools(&self, tools: Vec<ToolDefinition>) -> Self;
    fn model_name(&self) -> &str;
}

Expand description

Unified interface for LLM providers.

This trait provides a common abstraction over different LLM providers (Anthropic, OpenAI, Ollama, etc.), allowing provider-agnostic code.

§Example

use juncture::llm::{ChatModel, MockChatModel};
use juncture::Message;

let model = MockChatModel::new("gpt-4").with_response("Hello!");
let messages = vec![Message::human("Hi")];

let response = model.invoke(&messages, None).await?;

Required Methods§

Source

fn invoke<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, messages: &'life1 [Message], options: Option<&'life2 CallOptions>, ) -> Pin<Box<dyn Future<Output = Result<Message, LlmError>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Invoke the model and get a complete response.

This is the simplest way to use an LLM - send messages and get back a complete response. For real-time streaming, use Self::stream.

§Errors

Returns LlmError if:

Authentication fails
Network issues occur
The provider returns an invalid response
Rate limits are exceeded
The context length is exceeded

Source

fn stream( &self, messages: &[Message], options: Option<&CallOptions>, ) -> BoxStream<'_, Result<MessageChunk, LlmError>>

Stream the model response.

Returns a stream of MessageChunk values, allowing real-time processing of the model’s output as it’s generated.

§Example

use juncture::llm::ChatModel;
use futures::stream::StreamExt;

while let Some(chunk) = stream.next().await {
    let chunk = chunk?;
    // Process chunk
}

§Errors

The stream may yield LlmError values if errors occur during streaming.

Source

fn bind_tools(&self, tools: Vec<ToolDefinition>) -> Self

Bind tools to the model for function calling.

Returns a new instance of the model with the specified tools available. The model can then call these tools during inference when appropriate.

§Example

use juncture::llm::{ChatModel, MockChatModel, ToolDefinition};
use serde_json::json;

let model = MockChatModel::new("gpt-4");
let tools = vec![
    ToolDefinition {
        name: "get_weather".to_string(),
        description: "Get current weather".to_string(),
        parameters: json!({
            "type": "object",
            "properties": {
                "location": {"type": "string"}
            }
        }),
    },
];
let model_with_tools = model.bind_tools(tools);