openrouter-batch

A Rust client for the OpenRouter Batch API, plus synchronous structured chat completions and embeddings. Every wire behavior in this crate was verified against the live API.

Features

• Batch API (/beta/batches): submit chat-completions batches, poll to a terminal status, decode per-line results. One malformed result line never fails a completed batch — it arrives as BatchResultLine::Undecodable with the raw JSON preserved.
• Structured output: strict JSON Schema (response_format with strict: true), shared by the batch and synchronous paths so their wire shapes cannot drift. One schema per batch, as required by Google-backed routes.
• Embeddings (/v1/embeddings): array input, decoded back into input-ordered vectors with envelope/count/index/duplicate validation.
• Typed transport errors with retry classification (timeouts, connect failures, 408/429, 5xx are retryable) and provider request IDs surfaced on every failure.
• Testable seam: BatchTransport and EmbeddingTransport traits let you fake the API in your own tests; HttpTransport takes a configurable API root.

Wire rules encoded (live-verified)

• One batch carries one endpoint, one batch-level model, and one inline requests array; endpoint and model must serialize before requests (the API stream-parses and answers 400 otherwise).
• Submit answers 202 Accepted with status: "validating" — queued, not complete. Polls can 404 in the window after submit (eventual consistency).
• Terminal statuses: completed, failed, expired, cancelled. A completed batch can still contain per-request failures: exactly one of response or error is populated per result line.
• custom_id is capped at 64 characters; overlong keys are rejected locally before reaching the wire.
• The batch-level model must be a :batch variant slug.
• Server-side retention is 30 days: persist every result at poll time.

Usage

Structured batch

use openrouter_batch::{submit_structured, BatchStatus, BatchTransport, HttpTransport, Message, StructuredParams};
use serde_json::json;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let transport = HttpTransport::new("https://openrouter.ai/api", std::env::var("OPENROUTER_API_KEY")?.as_str())?;

    let params = StructuredParams {
        schema: json!({
            "type": "object",
            "properties": {"answer": {"type": "string"}},
            "required": ["answer"],
            "additionalProperties": false
        }),
        schema_name: "demo".to_string(),
        max_tokens: 300,
        disable_reasoning: false,
        require_parameters: false,
        reasoning_max_tokens: None,
    };

    let items = vec![
        ("req-1".to_string(), vec![Message::user("What is 2+2? State the number.")]),
        ("req-2".to_string(), vec![Message::user("What is 3*7? State the number.")]),
    ];

    let submitted = submit_structured(&transport, "google/gemini-2.5-flash-lite:batch", "idempotency-key", &params, items).await?;

    let mut consecutive_not_found = 0;
    let batch = loop {
        tokio::time::sleep(std::time::Duration::from_secs(5)).await;
        match transport.get(&submitted.id).await {
            Err(error) if error.not_found() => {
                consecutive_not_found += 1;
                if consecutive_not_found > 12 {
                    return Err(error.into());
                }
            }
            Err(error) => return Err(error.into()),
            Ok(batch) if batch.status.is_terminal() => break batch,
            Ok(_) => continue,
        }
    };

    for line in batch.results.unwrap_or_default() {
        if let (Some(id), Ok(content)) = (line.custom_id(), line.chat_content()) {
            println!("{id}: {content}");
        }
    }
    Ok(())
}

Synchronous structured completion

let transport = HttpTransport::with_timeout(
    "https://openrouter.ai/api",
    &api_key,
    std::time::Duration::from_secs(300),
)?;
let response = transport.chat_completion(&model, &params, vec![Message::user(prompt)]).await?;
let content = response.content()?;

Embeddings

use openrouter_batch::{EmbeddingTransport, HttpTransport, embedding_vectors};

let transport = HttpTransport::new(root, key)?;
let raw = transport.embed(&model, &inputs, &idempotency_key).await?;
let vectors = embedding_vectors(&raw.raw, inputs.len())?;
assert_eq!(vectors.len(), inputs.len());

Live smoke test CLI

cargo run --release --example cli -- structured
cargo run --release --example cli -- embeddings --input "first text" --input "second text"
cargo run --release --example cli -- batch

# Resume polling a batch after a `poll timeout`; the batch keeps
# processing server-side and results stay retrievable for 30 days.
cargo run --release --example cli -- poll --batch-id <batch-id>

The CLI resolves the API key from --api-key, then OPENROUTER_API_KEY, then a .env file in the current directory.

Notes

• The default request timeout is 30s, sized for batch calls. Synchronous chat completions can run far longer — use [HttpTransport::with_timeout].
• The Batch API lives under /beta/batches and may change; the client treats the API root as configuration, so pointed tests and fakes stay easy.
• Reasoning fields are opt-in: some routes make reasoning mandatory and fail every item when "reasoning": {"enabled": false} is present.

License

MIT