Crate llmprism 

llmprism is a Rust library for talking to Large Language Model (LLM) providers – OpenAI, Anthropic, Gemini, and ten more – through one consistent, fluent API, instead of learning each provider’s own SDK and JSON shape. It’s a Rust port of the PHP/Laravel library Prism.

A first example

use llmprism::Registry;

// Reads OPENAI_API_KEY / ANTHROPIC_API_KEY from the environment and registers
// whichever providers have a key set (gated by the crate features you enabled).
let registry = Registry::from_env();

let response = registry
    .text("openai", "gpt-4o-mini")?
    .with_prompt("Say hello in one word.")
    .generate()
    .await?;

println!("{}", response.text.unwrap_or_default());

Swap "openai" for "anthropic" and nothing else about this code needs to change – that consistency across providers is the reason this crate exists.

Streaming

Call stream instead of generate to get the reply incrementally, as a stream of StreamEvents, instead of waiting for the whole thing:

use futures::StreamExt;
use llmprism::{Registry, StreamEvent};

let registry = Registry::from_env();
let mut stream = registry
    .text("openai", "gpt-4o-mini")?
    .with_prompt("Count to three.")
    .stream();

while let Some(event) = stream.next().await {
    if let StreamEvent::TextDelta { text } = event? {
        print!("{text}");
    }
}

Tool calling works the same way here as it does for generate – if the model asks to call a tool, llmprism runs it, reports the result as a StreamEvent::ToolResult, and keeps streaming the model’s next round trip automatically. See stream_loop for how that’s implemented.

Structured output

Use Registry::structured instead of Registry::text to get a reply guaranteed to match a JSON shape you describe with schema, rather than free-form text:

use llmprism::schema::{BooleanSchema, ObjectSchema, Schema, StringSchema};
use llmprism::Registry;

let schema = ObjectSchema::new("review")
    .with_property(Schema::String(StringSchema::new("summary")), true)
    .with_property(Schema::Boolean(BooleanSchema::new("recommended")), true);

let registry = Registry::from_env();
let response = registry
    .structured("openai", "gpt-4o-mini", schema)?
    .with_prompt("Review this crate's ergonomics in one sentence.")
    .generate()
    .await?;

println!("{}", response.data);

response.data is a serde_json::Value you can deserialize into your own type with serde_json::from_value. See structured for how different providers get a model to comply with the schema under the hood.

How the pieces fit together

• Registry is where you register providers – or let Registry::from_env do it for you – and it’s the one place your application code asks for a provider by name. In tests, you register a testing::FakeProvider under that same name instead, so no other code has to know it’s being tested. (from_env isn’t linked above: it only exists once any provider feature is enabled, so a doc build with none enabled has nothing to link to.)
• Every capability – text generation (streaming or not), structured output, moderation, embeddings, reranking, images, and audio (text-to-speech and speech-to-text) – follows the same shape: ask the registry for a fluent request builder, chain .with_*() calls to describe what you want, then call an async method like text::PendingTextRequest::generate (or text::PendingTextRequest::stream for incremental output) to run it. Not every provider implements every capability – Anthropic, for instance, has no moderation endpoint – calling one it doesn’t returns Error::Unsupported rather than failing to compile.
• Tool is how you give the model something it can call on your behalf – a weather lookup, a database query, anything. Hand a list of tools to a text request and llmprism handles the entire back-and-forth: if the model asks to call a tool, the tool runs, its result goes back to the model, and this repeats until the model gives a final answer (see tool_loop).
• schema describes the shape of data – a tool’s arguments, or a structured-output request’s required shape – in a way every provider understands, without you hand-writing each provider’s own JSON Schema dialect.
• value_objects holds the plain data types shared by every provider and every capability: conversation messages, token usage, finish reasons, and so on.

Feature flags

Nothing is enabled by default. Each provider lives behind its own Cargo feature flag (openai, anthropic, groq, deepseek, mistral, xai, openrouter, perplexity, zai, ollama, voyageai, elevenlabs, gemini), so a binary that only uses one provider isn’t forced to compile in HTTP client code for the rest. Enable full to turn on every provider at once. groq through zai all live in providers::openai_compatible (not linked here since that module, like providers itself, only exists once one of its features is enabled) and cover Text generation only – see that module’s own docs (with the relevant feature enabled) for why.

Re-exports

pub use error::Error;

pub use provider::Provider;

pub use registry::Registry;

pub use stream_event::StreamEvent;

pub use tool::Tool;

Modules

approval

ApprovalHandler – a hook that gates a specific tool call on an external decision, instead of running it automatically.

audio

Audio – text-to-speech (turn text into spoken audio) and speech-to-text (transcribe spoken audio into text). Start with crate::Registry::text_to_speech or crate::Registry::speech_to_text.

client

Shared HTTP plumbing used by every network-based provider: building the underlying HTTP client and turning a provider’s error response into a typed Error. If you’re just using this crate rather than implementing a new provider, you shouldn’t need anything in this module directly.

embeddings

Embeddings – turn text into numeric vectors for similarity search, clustering, retrieval, and so on. Start with crate::Registry::embeddings.

error

The error types this crate returns. Almost everything that can fail – a provider call, a tool execution, decoding a response – ends up as an Error, so you generally only need to match on one type regardless of which capability or provider you’re calling.

images

Image generation. Start with crate::Registry::images.

mcp

MCP (Model Context Protocol) client support – connect to a remote MCP server, discover its tools, and use them exactly like any other llmprism Tool, with no protocol-specific code in your application. Built on the official rmcp SDK.

middleware

ProviderMiddleware – wraps a Provider to intercept its calls: transform a request before it’s sent, transform or replace a response after it comes back, or skip the call entirely (a cache hit, a policy rejection). Start with Registry::wrap.

moderation

Moderation – check text against a provider’s content-safety classifier. Start with crate::Registry::moderation.

persistence

ConversationStore and PersistenceMiddleware – save a conversation’s message history after each call and reload it before the next one, keyed by an opaque id, so a caller only has to send the newest turn instead of replaying the whole conversation every time.

provider

The Provider trait – the interface every LLM backend implements.

providers

Concrete Provider implementations. Each one lives behind its own Cargo feature flag (see the crate root docs for the full list), so enabling openai is what makes openai::OpenAiProvider available, and so on (not linked here since each submodule – and therefore its types – only exists once its own feature is enabled, which isn’t guaranteed just because this module itself compiles). Most applications don’t construct these directly – prefer Registry::from_env, which does it for you based on which environment variables are set.

registry

Registry – where providers are registered and looked up by name.

rerank

Reranking – given a query and a list of documents, score and sort them by relevance. Start with crate::Registry::rerank, which hands you a PendingRerankRequest to configure and run.

schema

Describing the shape of data in a way every provider understands.

stream_event

Streaming events – what a streaming Text request yields incrementally, instead of making you wait for the entire reply. See PendingTextRequest::stream.

stream_loop

The multi-step tool-calling loop for streaming Text generation – the streaming counterpart to crate::tool_loop.

structured

Structured output – ask the model for a reply matching a specific JSON shape (described with crate::schema) instead of free-form text. Start with crate::Registry::structured.

tenancy

RequestContext, TenantRegistry, and UsageTrackingMiddleware – the hooks an application builds per-tenant provider resolution and usage tracking on top of, without this crate knowing anything about how identity is actually established.

testing

Test helpers for code that uses this crate, so you can test your own application logic without making real network calls or spending real API credits. See FakeProvider for the main entry point.

text

Text generation – the “chat with a model, optionally letting it call tools” capability. Start with crate::Registry::text, which hands you a PendingTextRequest to configure and run.

tool

The Tool trait – how you give a model something it can call.

tool_loop

The multi-step tool-calling loop shared by every provider.

tracing_middleware

GenAiTracingMiddleware – instruments every Provider call with a tracing span following the OpenTelemetry Semantic Conventions for Generative AI (the gen_ai.* attribute namespace – still in Development status as of this writing, so expect it to keep evolving; this module uses the current names, including renames like gen_ai.system -> gen_ai.provider.name and gen_ai.usage.prompt_tokens -> gen_ai.usage.input_tokens).

value_objects

The plain data types shared across every provider and capability – conversation messages, tool calls, token usage, and so on. If you’re building a request by hand (rather than through PendingTextRequest’s builder methods) or inspecting a response in detail, this is where the pieces live.