Crate llm 

aether-llm

Multi-provider LLM abstraction layer for Rust. Write your code once, then swap between Anthropic, OpenAI, OpenRouter, Ollama, and more by changing a single string.

Table of Contents

• Quick start
• Examples
  • Conversation with a system prompt
  • Tool use
  • Switching providers
  • Direct provider construction
• Providers
• Documentation
• Key Types
• Feature Flags
• License

Quick start

Parse a "provider:model" string into a provider, build a context, and stream the response:

use llm::parser::ModelProviderParser;
use llm::types::IsoString;
use llm::{ChatMessage, ContentBlock, Context, LlmResponse, StreamingModelProvider};
use tokio_stream::StreamExt;

#[tokio::main]
async fn main() -> llm::Result<()> {
    let parser = ModelProviderParser::default();
    let (provider, _model) = parser.parse("anthropic:claude-sonnet-4-5-20250929").await?;

    let context = Context::new(
        vec![ChatMessage::User {
            content: vec![ContentBlock::text("Explain ownership in Rust in two sentences.")],
            timestamp: IsoString::now(),
        }],
        vec![],
    );

    let mut stream = provider.stream_response(&context);
    while let Some(Ok(event)) = stream.next().await {
        if let LlmResponse::Text { chunk } = event {
            print!("{chunk}");
        }
    }
    Ok(())
}

Examples

Conversation with a system prompt

use llm::types::IsoString;
use llm::{ChatMessage, ContentBlock, Context};

let context = Context::new(
    vec![
        ChatMessage::System {
            content: "You are a helpful assistant that responds in haiku.".into(),
            timestamp: IsoString::now(),
        },
        ChatMessage::User {
            content: vec![ContentBlock::text("What is Rust?")],
            timestamp: IsoString::now(),
        },
    ],
    vec![], // no tools
);

Tool use

Define tools with JSON Schema, then feed results back after execution:

use llm::types::IsoString;
use llm::{
    AssistantReasoning, ChatMessage, ContentBlock, Context, ToolCallResult, ToolDefinition,
};

let tools = vec![ToolDefinition {
    name: "get_weather".into(),
    description: "Get current weather for a city".into(),
    parameters: r#"{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}"#.into(),
    server: None,
}];

let mut context = Context::new(
    vec![ChatMessage::User {
        content: vec![ContentBlock::text("What's the weather in Tokyo?")],
        timestamp: IsoString::now(),
    }],
    tools,
);

// After streaming the response and executing the tool call,
// feed the result back into the context:
context.push_assistant_turn(
    "Let me check the weather.",
    AssistantReasoning::default(),
    vec![Ok(ToolCallResult {
        id: "call_1".into(),
        name: "get_weather".into(),
        arguments: r#"{"city":"Tokyo"}"#.into(),
        result: "72F, sunny".into(),
    })],
);
// Then call provider.stream_response(&context) again for the final answer.

Switching providers

ModelProviderParser accepts any supported "provider:model" string, so switching is a one-line change:

use llm::parser::ModelProviderParser;

let parser = ModelProviderParser::default();

// Cloud providers (need API keys in env)
let (provider, _) = parser.parse("anthropic:claude-sonnet-4-5-20250929").await.unwrap();
let (provider, _) = parser.parse("openai:gpt-4o").await.unwrap();
let (provider, _) = parser.parse("openrouter:moonshotai/kimi-k2").await.unwrap();

// Local models (no API key needed)
let (provider, _) = parser.parse("ollama:llama3.2").await.unwrap();
let (provider, _) = parser.parse("llamacpp").await.unwrap();

Direct provider construction

When you need fine-grained control (temperature, max tokens), construct the provider directly:

use llm::providers::anthropic::AnthropicProvider;
use llm::ProviderFactory;

let provider = AnthropicProvider::from_env().await
    .unwrap()
    .with_model("claude-sonnet-4-5-20250929")
    .with_temperature(0.7)
    .with_max_tokens(4096);

Providers

Provider	Example model string	Env var
Anthropic	anthropic:claude-sonnet-4-5-20250929	ANTHROPIC_API_KEY
OpenAI	openai:gpt-4o	OPENAI_API_KEY
OpenRouter	openrouter:moonshotai/kimi-k2	OPENROUTER_API_KEY
ZAI	zai:GLM-4.6	ZAI_API_KEY
AWS Bedrock	bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0	AWS credentials
Ollama	ollama:llama3.2	None (local)
Llama.cpp	llamacpp	None (local)

Documentation

Full API documentation is available on docs.rs.

Key entry points:

• StreamingModelProvider – the core trait all providers implement
• Context – conversation state management
• ChatMessage – message types for building conversations
• LlmResponse – streaming response events
• ModelProviderParser – parse "provider:model" strings into providers

Key Types

• StreamingModelProvider – Core trait for all LLM providers. Implement this to add a new provider.
• Context – Manages the message history, tool definitions, and reasoning effort sent to the model.
• ChatMessage – Message enum with variants for user, assistant, and tool call messages.
• ToolDefinition – Describes a tool the model can invoke (name, description, JSON schema).
• LlmModel – Catalog of known models with metadata (context window, capabilities).

Feature Flags

Feature	Description
bedrock	AWS Bedrock provider support
oauth	OAuth authentication (used by Codex provider)
codex	OpenAI Codex provider (implies oauth)

License

MIT

Re-exports

pub use catalog::LlmModel;

pub use error::ContextOverflowError;

pub use error::LlmError;

pub use error::Result;

pub use provider::LlmResponseStream;

pub use provider::ProviderFactory;

pub use provider::StreamingModelProvider;

pub use providers::codex::perform_codex_oauth_flow;

Modules

alloyed

catalog

Auto-generated model catalog with metadata for all known LLM models.

error

oauth

OAuth 2.0 authentication for LLM providers that require it.

parser

provider

providers

Built-in LLM provider implementations.

testing

Test utilities for code that depends on StreamingModelProvider.

types

Structs

AssistantReasoning

Reasoning metadata from an assistant response.

Context

Manages the conversation state sent to an LLM provider.

EncryptedReasoningContent

Opaque encrypted reasoning content from an LLM response.

TokenUsage

Token usage reported by a single LLM API response. Providers fill in only the dimensions they expose; the rest stay None.

ToolCallError

Error result of a tool call

ToolCallRequest

Tool call request from the LLM

ToolCallResult

Successful result of a tool call

ToolDefinition

A tool that the LLM can invoke during generation.

Enums

ChatMessage

A message in the conversation history.

ContentBlock

A block of content within a ChatMessage::User message.

LlmResponse

A streaming response event from an LLM provider.

ProviderCredential

Credential for an LLM provider (e.g., Anthropic, OpenRouter)

ReasoningEffort

StopReason

The reason an LLM stopped generating.