## llmoxide
Rust SDK for **OpenAI**, **Anthropic**, **Gemini**, and **Ollama** with a small, shared surface:
non-streaming and streaming calls, plus normalized SSE `Event`s.
Lower-level types (`Client`, `ResponseRequest`, chat sessions, and the like) are documented in
Rustdoc (`cargo doc --open` in this repo); this page only shows the shortest path.
### Dependencies
```toml
[dependencies]
llmoxide = "0.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```
### Usage
Crate-root helpers return a `Prompt` handle. **`.send`**, **`.stream`**, and **`.list_models`**
work on every provider the same way.
**Anthropic**
`send`:
```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let key = std::env::var("ANTHROPIC_API_KEY")?;
let resp = llmoxide::anthropic(&key).send("Write a haiku about Rust.").await?;
if let Some(t) = resp.text() {
println!("{t}");
}
Ok(())
}
```
`stream`:
```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let key = std::env::var("ANTHROPIC_API_KEY")?;
llmoxide::anthropic(&key)
.stream("Write a haiku about Rust.", |ev| {
if let llmoxide::Event::TextDelta(t) = ev {
print!("{t}");
}
})
.await?;
Ok(())
}
```
Tools (crate **`llmoxide-tools`**) work the same on every provider; register schemas, then
`run_with_tools_text` on a `Prompt`. Use **`run_with_tools_stream_text`** when you want
**`llmoxide::Event::TextDelta`** / **`ToolCall`** callbacks during the loop (example:
`crates/llmoxide-tools/examples/add_tool_stream.rs`).
Debugging:
- Set **`LLMOXIDE_DEBUG_TOOLS_STREAM=1`** for stderr logs from the tools runner (rounds, tool-call
counts, response summaries).
- Set **`LLMOXIDE_DEBUG_ANTHROPIC_STREAM=1`** (or reuse `LLMOXIDE_DEBUG_TOOLS_STREAM=1`) for stderr
logs from the Anthropic SSE parser (tool JSON assembly, parse failures, fallthrough).
```toml
[dependencies]
llmoxide = "0.1"
llmoxide-tools = "0.1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
serde = { version = "1", features = ["derive"] }
schemars = { version = "0.8", default-features = false, features = ["derive"] }
```
```rust
use llmoxide_tools::{RunConfig, ToolMeta, ToolRegistry, ToolRunnerText};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Deserialize, JsonSchema)]
struct AddArgs {
a: i64,
b: i64,
}
#[derive(Debug, Clone, Serialize, JsonSchema)]
struct AddResult {
sum: i64,
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let key = std::env::var("ANTHROPIC_API_KEY")?;
let mut tools = ToolRegistry::new();
tools.register::<AddArgs, AddResult, _, _>(
ToolMeta::new("add").description("Add two integers."),
|args| async move { Ok(AddResult { sum: args.a + args.b }) },
);
let resp = llmoxide::anthropic(&key)
.run_with_tools_text(
"Use the add tool to compute 19 + 23. Reply with the number only.",
&tools,
RunConfig { max_rounds: 4 },
)
.await?;
println!("{}", resp.text().unwrap_or_default());
Ok(())
}
```
**OpenAI**
`send`:
```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let key = std::env::var("OPENAI_API_KEY")?;
let resp = llmoxide::openai(&key).send("Write a haiku about Rust.").await?;
if let Some(t) = resp.text() {
println!("{t}");
}
Ok(())
}
```
**Gemini**
```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let key = std::env::var("GEMINI_API_KEY")?;
let resp = llmoxide::gemini(&key).send("Write a haiku about Rust.").await?;
if let Some(t) = resp.text() {
println!("{t}");
}
Ok(())
}
```
**Ollama** (default `http://localhost:11434`)
```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let resp = llmoxide::ollama().send("Write a haiku about Rust.").await?;
if let Some(t) = resp.text() {
println!("{t}");
}
Ok(())
}
```
**Ollama** (custom base URL)
```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let resp = llmoxide::ollama_at("http://127.0.0.1:11434")
.send("Write a haiku about Rust.")
.await?;
if let Some(t) = resp.text() {
println!("{t}");
}
Ok(())
}
```
**List models**
```rust
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let key = std::env::var("ANTHROPIC_API_KEY")?;
let _models = llmoxide::anthropic(&key).list_models().await?;
Ok(())
}
```
Chain `.model("model-id")` or `.max_output_tokens(n)` before `.send` / `.stream` when you need
them. With the default `new_auto()` behavior, the first model from the provider list is used when
you do not set `.model`; if that does not match your account (for example some Anthropic setups),
set `.model("…")` explicitly or pick an id from `.list_models().await`.
### Multimodal messages and extensions
`Message` content is a list of [`ContentPart`](https://docs.rs/llmoxide/latest/llmoxide/enum.ContentPart.html)
values. Besides plain `Text` and tool parts, the enum includes **images** (`ImageUrl`,
`ImageBase64`), **reasoning text** (`Thinking`, mapped to native formats where the provider supports
it), and an **opaque** `Citation` payload for references. Provider adapters decide what can be sent
on each turn (for example Ollama attaches base64 images on **user** messages only; `ImageUrl` may be
unsupported on some backends).
For response-time fields that do not fit a stable shape yet (citations, safety metadata, usage
details), check **`Response::metadata`** (and optionally the `raw-json` feature for the full provider
payload). The API stays **`#[non_exhaustive]`** so new variants and metadata keys can appear without
forcing a major bump for every provider addition.
### Stability / compatibility
This repo is still early-stage. Until `1.0`:
- **Public API may change** between minor versions (`0.x`).
- Provider streaming details are best-effort; the crate aims to keep the *high-level* contract
stable (e.g. `Response::text()`, tool calling loop behavior), while adapters may need updates as
providers evolve.
Tool-calling + streaming invariants (`llmoxide-tools`):
- `run_with_tools_stream_text` emits **exactly one** final `Event::Completed` (for the last
assistant message).
- Tool-calling rounds may have **empty assistant text** (e.g. a turn that only emits a `tool_use`
block).
### CLI
Binary name: **`llmoxide`** (package **`llmoxide-cli`**).
```bash
cargo install llmoxide-cli
```
Examples (`-a` Anthropic, `-o` OpenAI, `-g` Gemini, `-l` Ollama):
```bash
ANTHROPIC_API_KEY=... llmoxide -a send "Write a haiku about Rust."
ANTHROPIC_API_KEY=... llmoxide -a chat
OPENAI_API_KEY=... llmoxide -o send "Write a haiku about Rust."
OPENAI_API_KEY=... llmoxide -o stream "Write a haiku about Rust."
GEMINI_API_KEY=... llmoxide -g send "Write a haiku about Rust."
llmoxide -l send "Write a haiku about Rust."
```
See `llmoxide --help` for `stream`, `chat`, `models`, and flags like `--model` and `--ollama-host`.
### Live integration tests (local keys)
Opt-in network tests (`#[ignore]`). Use a local `.env` (see `.env.example`), then:
```bash
cargo test --test live_providers -- --ignored --nocapture
cargo test -p llmoxide-tools --test live_add_tools -- --ignored --nocapture
```
The second command exercises **tool calling** (`add`) against OpenAI, Anthropic, Gemini, and Ollama (set `LLMOXIDE_TEST_OLLAMA=1` or `OLLAMA_HOST` for local Ollama; use a model that supports tools). It also runs **`live_openai_add_tool_stream`** and **`live_anthropic_add_tool_stream`** (`run_with_tools_stream_text`).