aether-llm

Multi-provider LLM abstraction layer for Rust. Write your code once, then swap between Anthropic, OpenAI, OpenRouter, Ollama, and more by changing a single string.

Table of Contents

• Quick start
• Examples
  • Conversation with a system prompt
  • Tool use
  • Switching providers
  • Direct provider construction
• Providers
• Documentation
• Key Types
• Feature Flags
• License

Quick start

Parse a "provider:model" string into a provider, build a context, and stream the response:

use llm::parser::ModelProviderParser;
use llm::types::IsoString;
use llm::{
    ChatMessage, ContentBlock, Context, LlmResponse, MessageId, ProviderConnectionOverrides, StreamingModelProvider,
};
use tokio_stream::StreamExt;

#[tokio::main]
async fn main() -> llm::Result<()> {
    let parser = ModelProviderParser::default();
    let connections = ProviderConnectionOverrides::default();
    let (provider, _model) = parser.parse("anthropic:claude-sonnet-4-5-20250929", &connections).await?;

    let context = Context::new(
        vec![ChatMessage::User {
            message_id: MessageId::new(),
            content: vec![ContentBlock::text("Explain ownership in Rust in two sentences.")],
            timestamp: IsoString::now(),
        }],
        vec![],
    );

    let mut stream = provider.stream_response(&context);
    while let Some(Ok(event)) = stream.next().await {
        if let LlmResponse::Text { chunk } = event {
            print!("{chunk}");
        }
    }
    Ok(())
}

Examples

Conversation with a system prompt

use llm::types::IsoString;
use llm::{ChatMessage, ContentBlock, Context, MessageId};

let context = Context::new(
    vec![
        ChatMessage::System {
            content: "You are a helpful assistant that responds in haiku.".into(),
            timestamp: IsoString::now(),
        },
        ChatMessage::User {
            message_id: MessageId::new(),
            content: vec![ContentBlock::text("What is Rust?")],
            timestamp: IsoString::now(),
        },
    ],
    vec![], // no tools
);

Tool use

Define tools with JSON Schema, then feed results back after execution:

use llm::types::IsoString;
use llm::{
    AssistantContent, AssistantReply, ChatMessage, ContentBlock, Context, MessageId, ToolCallRequest, ToolCallResult,
    ToolDefinition,
};

let tools = vec![ToolDefinition::new(
    "get_weather",
    "Get current weather for a city",
    serde_json::json!({
        "type": "object",
        "properties": { "city": { "type": "string" } },
        "required": ["city"]
    }),
)];

let mut context = Context::new(
    vec![ChatMessage::User {
        message_id: MessageId::new(),
        content: vec![ContentBlock::text("What's the weather in Tokyo?")],
        timestamp: IsoString::now(),
    }],
    tools,
);

// After streaming the response and executing the tool call,
// feed the result back into the context:
let arguments = r#"{"city":"Tokyo"}"#;
let reply = AssistantReply::from(vec![
    AssistantContent::text("Let me check the weather."),
    AssistantContent::ToolCall(ToolCallRequest {
        id: "call_1".into(),
        name: "get_weather".into(),
        arguments: arguments.into(),
    }),
]);
context.push_assistant_turn(
    MessageId::new(),
    "anthropic:claude-sonnet-4-5-20250929".parse().ok(),
    reply,
    vec![Ok(ToolCallResult {
        id: "call_1".into(),
        name: "get_weather".into(),
        arguments: arguments.into(),
        result: "72F, sunny".into(),
    })],
);
// Then call provider.stream_response(&context) again for the final answer.

Switching providers

ModelProviderParser accepts any supported "provider:model" string, so switching is a one-line change:

# async fn example() {
use llm::parser::ModelProviderParser;
use llm::ProviderConnectionOverrides;

let parser = ModelProviderParser::default();
let connections = ProviderConnectionOverrides::default();

// Cloud providers (need API keys in env)
let (provider, _) = parser.parse("anthropic:claude-sonnet-4-5-20250929", &connections).await.unwrap();
let (provider, _) = parser.parse("openai:gpt-4o", &connections).await.unwrap();
let (provider, _) = parser.parse("openrouter:moonshotai/kimi-k2", &connections).await.unwrap();

// Local models (no API key needed)
let (provider, _) = parser.parse("ollama:llama3.2", &connections).await.unwrap();
let (provider, _) = parser.parse("llamacpp", &connections).await.unwrap();
# }

Direct provider construction

When you need fine-grained control, construct the provider directly and configure request settings on the context:

# fn example() {
use llm::providers::generic::{ANTHROPIC, GenericProvider};
use llm::{Context, ModelSettings};

let provider = GenericProvider::from_env(&ANTHROPIC)
    .unwrap()
    .with_model("claude-sonnet-4-5-20250929");

let mut context = Context::new(Vec::new(), Vec::new());
context.set_model_settings(ModelSettings {
    temperature: Some(0.7),
    max_tokens: Some(4096),
    ..ModelSettings::default()
});
# }

Providers

Documentation

Full API documentation is available on docs.rs.

Key entry points:

• [StreamingModelProvider] -- the core trait all providers implement
• [Context] -- conversation state management
• [ChatMessage] -- message types for building conversations
• [LlmResponse] -- streaming response events
• ModelProviderParser -- parse "provider:model" strings into providers

Key Types

• StreamingModelProvider -- Core trait for all LLM providers. Implement this to add a new provider.
• Context -- Manages the message history, tool definitions, and reasoning effort sent to the model.
• ChatMessage -- Message enum with variants for user, assistant, and tool call messages.
• ToolDefinition -- Describes a tool the model can invoke (name, description, JSON schema).
• LlmModel -- Catalog of known models with metadata (context window, capabilities).

Feature Flags

License

MIT