llmoxide 0.1.0

Provider-agnostic Rust SDK for OpenAI, Anthropic, Gemini, and Ollama (streaming + tools)
Documentation

llmoxide

Rust SDK for OpenAI, Anthropic, Gemini, and Ollama with a small, shared surface: non-streaming and streaming calls, plus normalized SSE Events.

Lower-level types (Client, ResponseRequest, chat sessions, and the like) are documented in Rustdoc (cargo doc --open in this repo); this page only shows the shortest path.

Dependencies

[dependencies]
llmoxide = "0.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Usage

Crate-root helpers return a Prompt handle. .send, .stream, and .list_models work on every provider the same way.

Anthropic

send:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("ANTHROPIC_API_KEY")?;
    let resp = llmoxide::anthropic(&key).send("Write a haiku about Rust.").await?;
    if let Some(t) = resp.text() {
        println!("{t}");
    }
    Ok(())
}

stream:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("ANTHROPIC_API_KEY")?;
    llmoxide::anthropic(&key)
        .stream("Write a haiku about Rust.", |ev| {
            if let llmoxide::Event::TextDelta(t) = ev {
                print!("{t}");
            }
        })
        .await?;
    Ok(())
}

Tools (crate llmoxide-tools) work the same on every provider; register schemas, then run_with_tools_text on a Prompt. Use run_with_tools_stream_text when you want llmoxide::Event::TextDelta / ToolCall callbacks during the loop (example: crates/llmoxide-tools/examples/add_tool_stream.rs).

Debugging:

  • Set LLMOXIDE_DEBUG_TOOLS_STREAM=1 for stderr logs from the tools runner (rounds, tool-call counts, response summaries).
  • Set LLMOXIDE_DEBUG_ANTHROPIC_STREAM=1 (or reuse LLMOXIDE_DEBUG_TOOLS_STREAM=1) for stderr logs from the Anthropic SSE parser (tool JSON assembly, parse failures, fallthrough).
[dependencies]
llmoxide = "0.1"
llmoxide-tools = "0.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
serde = { version = "1", features = ["derive"] }
schemars = { version = "0.8", default-features = false, features = ["derive"] }

use llmoxide_tools::{RunConfig, ToolMeta, ToolRegistry, ToolRunnerText};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Deserialize, JsonSchema)]
struct AddArgs {
    a: i64,
    b: i64,
}

#[derive(Debug, Clone, Serialize, JsonSchema)]
struct AddResult {
    sum: i64,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("ANTHROPIC_API_KEY")?;
    let mut tools = ToolRegistry::new();
    tools.register::<AddArgs, AddResult, _, _>(
        ToolMeta::new("add").description("Add two integers."),
        |args| async move { Ok(AddResult { sum: args.a + args.b }) },
    );
    let resp = llmoxide::anthropic(&key)
        .run_with_tools_text(
            "Use the add tool to compute 19 + 23. Reply with the number only.",
            &tools,
            RunConfig { max_rounds: 4 },
        )
        .await?;
    println!("{}", resp.text().unwrap_or_default());
    Ok(())
}

OpenAI

send:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("OPENAI_API_KEY")?;
    let resp = llmoxide::openai(&key).send("Write a haiku about Rust.").await?;
    if let Some(t) = resp.text() {
        println!("{t}");
    }
    Ok(())
}

Gemini

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("GEMINI_API_KEY")?;
    let resp = llmoxide::gemini(&key).send("Write a haiku about Rust.").await?;
    if let Some(t) = resp.text() {
        println!("{t}");
    }
    Ok(())
}

Ollama (default http://localhost:11434)

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let resp = llmoxide::ollama().send("Write a haiku about Rust.").await?;
    if let Some(t) = resp.text() {
        println!("{t}");
    }
    Ok(())
}

Ollama (custom base URL)

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let resp = llmoxide::ollama_at("http://127.0.0.1:11434")
        .send("Write a haiku about Rust.")
        .await?;
    if let Some(t) = resp.text() {
        println!("{t}");
    }
    Ok(())
}

List models

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let key = std::env::var("ANTHROPIC_API_KEY")?;
    let _models = llmoxide::anthropic(&key).list_models().await?;
    Ok(())
}

Chain .model("model-id") or .max_output_tokens(n) before .send / .stream when you need them. With the default new_auto() behavior, the first model from the provider list is used when you do not set .model; if that does not match your account (for example some Anthropic setups), set .model("…") explicitly or pick an id from .list_models().await.

Multimodal messages and extensions

Message content is a list of ContentPart values. Besides plain Text and tool parts, the enum includes images (ImageUrl, ImageBase64), reasoning text (Thinking, mapped to native formats where the provider supports it), and an opaque Citation payload for references. Provider adapters decide what can be sent on each turn (for example Ollama attaches base64 images on user messages only; ImageUrl may be unsupported on some backends).

For response-time fields that do not fit a stable shape yet (citations, safety metadata, usage details), check Response::metadata (and optionally the raw-json feature for the full provider payload). The API stays #[non_exhaustive] so new variants and metadata keys can appear without forcing a major bump for every provider addition.

Stability / compatibility

This repo is still early-stage. Until 1.0:

  • Public API may change between minor versions (0.x).
  • Provider streaming details are best-effort; the crate aims to keep the high-level contract stable (e.g. Response::text(), tool calling loop behavior), while adapters may need updates as providers evolve.

Tool-calling + streaming invariants (llmoxide-tools):

  • run_with_tools_stream_text emits exactly one final Event::Completed (for the last assistant message).
  • Tool-calling rounds may have empty assistant text (e.g. a turn that only emits a tool_use block).

CLI

Binary name: llmoxide (package llmoxide-cli).

cargo install llmoxide-cli

Examples (-a Anthropic, -o OpenAI, -g Gemini, -l Ollama):

ANTHROPIC_API_KEY=... llmoxide -a send "Write a haiku about Rust."
ANTHROPIC_API_KEY=... llmoxide -a chat
OPENAI_API_KEY=... llmoxide -o send "Write a haiku about Rust."
OPENAI_API_KEY=... llmoxide -o stream "Write a haiku about Rust."
GEMINI_API_KEY=... llmoxide -g send "Write a haiku about Rust."
llmoxide -l send "Write a haiku about Rust."

See llmoxide --help for stream, chat, models, and flags like --model and --ollama-host.

Live integration tests (local keys)

Opt-in network tests (#[ignore]). Use a local .env (see .env.example), then:

cargo test --test live_providers -- --ignored --nocapture
cargo test -p llmoxide-tools --test live_add_tools -- --ignored --nocapture

The second command exercises tool calling (add) against OpenAI, Anthropic, Gemini, and Ollama (set LLMOXIDE_TEST_OLLAMA=1 or OLLAMA_HOST for local Ollama; use a model that supports tools). It also runs live_openai_add_tool_stream and live_anthropic_add_tool_stream (run_with_tools_stream_text).