llmleaf-client (Rust)

Async Rust SDK for the llmleaf LLM proxy.

llmleaf speaks OpenAI/OpenRouter-shaped JSON over HTTP, so the public types are plain serde structs and the transport is reqwest over rustls (no system OpenSSL). The proto is the source of truth: build.rs compiles it with prost-build on every build and exposes the messages under pb as a codegen check — the serde types at the crate root are what you actually use.

Install

This crate is its own standalone workspace inside the monorepo (intentionally not a member of the root workspace). Depend on it by path or git:

[dependencies]
llmleaf-client = { git = "https://github.com/codefionn/llmleaf", branch = "main" }
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
futures = "0.3"

You need protoc (libprotoc 35) on PATH at build time — prost shells out to it.

Example

use futures::StreamExt;
use llmleaf_client::{Client, ChatRequest, ChatMessage};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new("https://gateway.example.com", "sk-...")?;

    // Non-streaming chat.
    let resp = client
        .chat(ChatRequest::new("gpt-4o-mini", vec![ChatMessage::user("Say hi.")]))
        .await?;
    println!("{}", resp.first_text().unwrap_or_default());

    // Streaming chat — yields chunks, stops on the [DONE] sentinel.
    let mut stream = client
        .chat_stream(ChatRequest::new("gpt-4o-mini", vec![ChatMessage::user("Count to 5.")]))
        .await?;
    while let Some(chunk) = stream.next().await {
        if let Some(delta) = chunk?.first_delta_text() {
            print!("{delta}");
        }
    }
    Ok(())
}

OpenAI Responses dialect

POST /v1/responses is the same canonical core behind a different edge dialect. input is a bare string (one user message) or an array of items; use ResponseItem to build multi-turn input (messages, function_call / function_call_output replay, reasoning items). llmleaf is stateless, so the response always reports "store": false.

use futures::StreamExt;
use llmleaf_client::{ResponsesRequest, ResponseItem};

// Non-streaming — bare-string input.
let resp = client
    .responses(ResponsesRequest::new("gpt-4o-mini", "Say hi."))
    .await?;
println!("{}", resp.output_text());

// Multi-turn item array (replaying a tool call and its result).
let req = ResponsesRequest::new(
    "gpt-4o-mini",
    vec![
        ResponseItem::user("What's the weather in Paris?"),
        ResponseItem::function_call("call_1", "get_weather", r#"{"city":"Paris"}"#),
        ResponseItem::function_call_output("call_1", r#"{"temp_c":21}"#),
    ],
);

// Streaming — typed events, NO [DONE] sentinel: the stream ends on the terminal
// response.completed / .incomplete / .failed event. Accumulate output_text deltas.
let mut events = client.responses_stream(req).await?;
while let Some(event) = events.next().await {
    if let Some(delta) = event?.output_text_delta() {
        print!("{delta}");
    }
}

Need a timeout, an admin token, or your own reqwest::Client? Use the builder:

use std::time::Duration;

let client = Client::builder("https://gateway.example.com", "sk-...")
    .timeout(Duration::from_secs(30))
    .admin_token("admin-...")   // adds per-model `endpoints` to GET /v1/models
    .build()?;

Endpoints

Gateway errors come back as Error::Api { status, message }, parsed from {"error":{"message":"..."}}.

Run the example

export LLMLEAF_BASE_URL="https://gateway.example.com"
export LLMLEAF_API_KEY="sk-..."
cargo run --example basic

It lists models, does a non-streaming chat, streams one, then does the same pair over the Responses dialect (printing deltas live).

Regenerate from the proto

Codegen is wired into build.rs, so a build is a regeneration — after editing ../proto/llmleaf/v1/llmleaf.proto, just rebuild (cargo build, or make gen-rust from clients/).

Notes

• The pb (prost) types are the codegen proof, not the wire types — use the serde types at the crate root for anything over HTTP.
• Free-form JSON fields (extra, ResponseFormat.json_schema, …) are spliced verbatim and not validated (intentional passthrough); ChatRequest.extra keys merge at the top level.
• No retries/backoff. The realtime WebSocket surface is out of scope.

License

Dual-licensed under Apache-2.0 or MIT at your option. Copyright (C) 2026 Fionn Langhans.