Crate allama

Expand description

Low-level, provider-agnostic LLM abstractions for Rust.

allama is the low-level layer of an LLM stack: portable request and response types, streaming, and provider traits, implemented by provider crates (allama-openai, allama-anthropic, allama-gemini, …). It is a building block, not an agent framework: there are no agent loops here.

Status: pre-1.0, so a minor release may break the API; every allama-* crate shares one version.

§Quick start

Pick the provider and model at runtime from one string, a model address such as openai:gpt-5.5, with the allama-providers crate:

[dependencies]
allama = "0.1"
allama-providers = "0.1"
anyhow = "1"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

use allama::prelude::*;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Reads OPENAI_API_KEY. Try "anthropic:claude-haiku-4-5",
    // "gemini:gemini-3.5-flash" or "ollama:qwen3:1.7b".
    let (provider, model) = allama_providers::chat::from_model("openai:gpt-5.5")?;

    let request = ChatRequest::new(model).user("Explain Rust ownership in two sentences.");
    let response = provider.chat(&request).await?;
    println!("{}", response.text_or_empty());
    Ok(())
}

main returns anyhow::Result so that a failure prints the provider’s message, which Error’s Debug withholds.

To print the answer as it is generated, stream it. The collector builds the same ChatResponse that chat returns:

use allama::prelude::*;

async fn stream(provider: &DynChatProvider, request: &ChatRequest) -> Result<ChatResponse> {
    let mut stream = provider.chat_stream(request).await?.with_collector();
    while let Some(item) = stream.next_event().await {
        if let StreamEvent::TextDelta { text, .. } = item?.event {
            print!("{text}");
        }
    }
    stream.finish()
}

Before production, read the README’s “Before production” checklist. Providers from allama_providers wait out a provider’s Retry-After of up to 120 s by default, which suits batch work but not someone waiting (RetryPolicy::max_retry_after). The checklist also covers refusals and cut-off answers, reasoning under max_output_tokens, settings a model cannot take, warnings, and credentials.

The terms these docs use, such as the commit point, a model address, an attempt timeout or a deadline and a pending tool call, are defined in the glossary.

§Capabilities

Each model operation is a sibling trait over ProviderIdentity:

ChatProvider: one-shot and streaming chat.
EmbeddingProvider: batch text embedding.
DecisionProvider: typed questions answered with probabilities (decision models; see decision).

§Portability model

The portable core covers what genuinely aligns across providers. What does not align stays provider-specific, in three deliberate places:

Provider options (RequestOptions): each provider declares an options type; callers attach the exact type or namespaced JSON.
Provider data (ProviderData): namespaced, serializable state on content, messages, tools, and responses (signatures, item ids, …). The namespace is the data’s origin; adapters only replay their own.
Other content (OtherContent): blocks the portable model does not represent, preserved verbatim with their origin and provider data.

Adapters never drop portable features silently: they fail with Error::Unsupported or report a Warning (see UnsupportedPolicy).

§Modules

wrappers: retry, fallback, timeout, model-mapping, request-transform, recording, and tracing wrappers.
decision: decision-model questions and answers, and typed decisions (derives with feature derive).
chat and embedding: an overview of the chat and embedding types, which are exported at the crate root (see Paths); chat::stream holds stream collection and commit-point helpers.
adapter: the provider-author toolkit, such as the prepare step providers run before sending, which makes validation (Validation) a dry run of the call, and the JSON Schema passes.
warning: warnings, the unsupported-feature policy, and warning events.
extract (feature extract): typed structured extraction.
provider_data: provider-scoped data and replay rules.
history: versioned conversation history for persistence, and tool-call validation before sending.
mock (feature mock): test doubles.

§Paths

The crate root is the path of every type it exports: allama::ChatRequest, allama::StreamEvent, allama::EmbeddingRequest. The prelude imports from it, and the examples and guides use it. The modules hold only what the root does not export: the stream helpers in chat::stream (until_committed, StreamCollector, StreamTranscript, …) and the decision questions and answers in decision (Question, Answer, Decision, …). Two kinds of root type keep a second path: a few are defined and documented in a module (StreamEvent in chat::stream, Warning in warning; import them from the root all the same), and decision re-exports the root decision types (DecisionRequest, DecisionProvider, …) for code that imports them alongside the questions and answers. Provider-author types live only in adapter.

§Cargo features

The core stays lean: every feature but warning-events is off by default.

Feature	Enables	Pulls in
`warning-events` (default)	Every warning a call reports is also a `tracing` event (`warning::emit`), deduplicated by its code, with a target per kind; `RetryingProvider` retries are `allama::retry` events and `FallbackProvider` failovers `allama::fallback` events	`tracing`
`schema`	`allama::schemars`, `ToolDefinition::from_schema`, `ResponseFormat::from_schema`, field paths in `ToolArgumentsError`	`schemars`, `serde_path_to_error`
`extract`	`allama::extract` (`ExtractExt::extract`, `Extractor`), also in the prelude; implies `schema`	`schemars`, `serde_path_to_error`
`derive`	`#[derive(Decision, ChoiceSet, Scale)]` in `decision`	`allama-derive` (`syn`, `quote`, `proc-macro2`, `heck`)
`mock`	`allama::mock`: `MockProvider`, `MockStreamingProvider`, `MockEmbeddingProvider`, `MockDecisionProvider`, and other test doubles	nothing
`tracing`	`wrappers::TracingProvider` (OpenTelemetry GenAI spans), with the resilience summary wrappers report through `wrappers::call_scope`, and retry and fallback attempt events	`tracing`

Deriving tool parameters from a Rust type (ToolDefinition::from_schema) needs schema:

allama = { version = "0.1", features = ["schema"] }

§Common tasks

Choosing a provider at runtime from a model address (openai:gpt-5.5) is allama_providers::chat::from_model; the web_backend example in allama-examples shows a whole chat turn (history, streaming, one tool round trip, errors by kind, usage). The tasks below build on it. For a web backend (storing conversations and the system prompt, an axum SSE handler, failures after the first token, rendering history, testing handlers with mocks), read the applications guide.

Map errors that occur before any output to HTTP status codes with chat::stream::until_committed: it holds a stream until its first model output, so a rate limit or an overload still becomes a 429 or 503 with Retry-After instead of an error event after 200 OK.

§A wrapper stack for providers built directly from adapter crates

A provider from allama_providers (a model address, or a provider you built and passed to allama_providers::chat::wrap) already retries, with a per-attempt timeout from its ProviderSettings: add only fallback, a deadline and tracing around it, as the applications guide’s provider stack does. Adding a RetryingProvider there would nest retries.

A provider built directly from an adapter crate (allama_openai, allama_anthropic, …) retries nothing. Wrap it once at startup. Outermost first: tracing, a deadline on the whole call, fallback, retry, a timeout per attempt, and model mapping around the provider it targets (see wrappers for why this order):

use std::time::Duration;

use allama::wrappers::{
    FallbackProvider, MapModelProvider, RetryPolicy, RetryingProvider, TimeoutPolicy,
    TimeoutProvider, TracingProvider,
};
use allama::{ChatProvider, DynChatProvider};

fn production(anthropic: DynChatProvider, openai: DynChatProvider) -> impl ChatProvider {
    // Bound each attempt; a timed-out attempt is retried. Three attempts
    // of 30 s and their backoff end inside the deadline below, with
    // about 30 s left for the fallback. A model whose calls take longer
    // needs both raised.
    let attempt = TimeoutPolicy::new()
        .request(Duration::from_secs(30))
        .stream_idle(Duration::from_secs(120));
    // Someone is waiting: a `Retry-After` longer than 5 s fails over at
    // once instead of holding the request.
    let retries = RetryPolicy::new().max_retry_after(Duration::from_secs(5));
    // A deadline across every attempt, backoff and failover, before any
    // output: the whole unary call, and a stream's wait for its first
    // output (never `stream_total`, which would cut off long answers).
    let deadline = TimeoutPolicy::new().request(Duration::from_secs(120));

    TracingProvider::new(TimeoutProvider::new(
        FallbackProvider::new(
            RetryingProvider::new(TimeoutProvider::new(anthropic, attempt), retries.clone()),
            RetryingProvider::new(TimeoutProvider::new(
                // Callers name Claude models; the OpenAI side gets its own.
                MapModelProvider::new(openai, |model: &str| match model {
                    "claude-opus-4-1" => "gpt-5.5".into(),
                    _ => "gpt-5-mini".into(),
                }),
                attempt,
            ), retries),
        ),
        deadline,
    ))
}

RetryingProvider retries retryable errors before any output has streamed, follows a server’s retry directive (x-should-retry) when it sends one, honors a provider’s Retry-After up to its max_retry_after (120 s by default, which suits batch work; the stack above sets 5 s for a server with someone waiting), and gives up on longer hints so the fallback takes over. Error::Configuration (a missing key, an unknown model address) is never retried or failed over.

§Cross-provider fallback

FallbackProvider sends the same request to its second provider when the first fails before producing output with an error another provider may not have (an outage, a rate limit, a request it cannot serve; not a bad request, a rejected key or a content filter: see should_fallback). Give the second side its own model ids with MapModelProvider, as above. Conversations survive the switch: opaque state such as thinking signatures is namespaced by its origin and only replayed to that provider. After a failover, the response says who served it and what failed first:

use allama::wrappers::FallbackAttempts;
use allama::ChatResponse;

fn report(response: &ChatResponse) -> allama::Result<()> {
    println!("served by {:?}", response.provider);
    if let Some(attempts) = response.provider_data.get::<FallbackAttempts>()? {
        for attempt in &attempts {
            println!("{} failed: {}", attempt.provider, attempt.error_kind);
        }
    }
    Ok(())
}

On failover, do nothing with the history: each adapter drops the state it cannot use for that request only. When a conversation moves to another provider for good, drop the old provider’s state once with History::retain_provider_state_for, passing the new provider.

§Extraction with cost accounting

An extraction may take several provider calls (retries are opt-in), and a failed one was still billed. Both outcomes carry the same ExtractionMetadata, with the usage of every pass summed; a failure also keeps the last response:

use allama::extract::Extractor;
use allama::schemars::{self, JsonSchema};
use allama::{ChatProvider, ChatRequest, Error, Usage};

#[derive(serde::Deserialize, JsonSchema)]
struct Invoice {
    total_cents: u64,
}

async fn invoice<P: ChatProvider>(
    provider: &P,
    request: &ChatRequest,
    bill: impl Fn(&Usage),
) -> allama::Result<Invoice> {
    let result = Extractor::new()
        .retries(1)
        .extract::<_, Invoice>(provider, request)
        .await;
    match &result {
        Ok(extracted) => extracted.metadata.usage.iter().for_each(&bill),
        Err(Error::Extract { failure, .. }) => {
            failure.metadata().usage.iter().for_each(&bill);
            if let Some(response) = failure.response() {
                eprintln!("unusable output: {}", response.text_or_empty());
            }
        }
        Err(_) => {}
    }
    result.map(|extracted| extracted.value)
}

For one call without recovery, provider.extract::<T>(&request) (ExtractExt) is enough.

§Embeddings at scale

Providers cap the inputs per request. embed_chunked splits a batch at the provider’s embedding_limits (or the size you pass when the provider reports none) and returns one response with the vectors in input order and the usage summed:

use allama::{EmbeddingProvider, EmbeddingProviderExt, EmbeddingRequest};

async fn index<P: EmbeddingProvider + Sync>(
    provider: &P,
    documents: Vec<String>,
) -> allama::Result<Vec<Vec<f32>>> {
    let request = EmbeddingRequest::new("text-embedding-3-small").inputs(documents);
    Ok(provider.embed_chunked(&request, 100).await?.embeddings)
}

Chunks are sent one after another, and one failed chunk fails the call. A provider from allama_providers already retries each chunk; wrap one built directly from an adapter crate in a RetryingProvider, as above, but never both, or retries nest. Split very large corpora yourself to checkpoint progress.

§Persisting conversations

Requests, responses, and messages serialize with serde. History adds a format version, so stored conversations load safely after the format evolves. A history stores the messages only; set the system prompt on every request, so a new prompt applies to old conversations:

use allama::history::History;
use allama::{ChatRequest, ChatResponse};

const SYSTEM_PROMPT: &str = "You are a helpful assistant.";

fn save(request: &ChatRequest, response: &ChatResponse) -> serde_json::Result<String> {
    let mut history = History::from_request(request);
    history.messages.push(response.to_assistant_message());
    serde_json::to_string(&history)
}

fn resume(stored: &str, model: &str, message: &str) -> serde_json::Result<ChatRequest> {
    let history: History = serde_json::from_str(stored)?;
    Ok(history.into_request(model).system(SYSTEM_PROMPT).user(message))
}

let stored = save(&first, &reply)?;
let next = resume(&stored, "demo-model", "Thanks")?;
assert_eq!(next.system.len(), 1);
assert_eq!(next.messages.len(), 3);

A conversation that keeps its own prompt, such as a persona the user picked, stores what it needs to rebuild the prompt (the text, or a persona id) next to the history, and builds the SystemPrompt (cache breakpoint, options) on every request. Message builders append; to replace the messages or the system prompts, assign the public field.

The assistant message keeps only the response content. To keep the response id, usage and the rest per turn, store ChatResponse::envelope with the message (see The turn envelope).

Provider state (signatures, encrypted reasoning) travels inside the messages, so a reloaded conversation continues exactly where it stopped, including on reasoning models. The application’s own data about a message (ids, timestamps, UI state) goes in its app_data, which is persisted but never sent (see Application data).

§Serialization and stability

Most public types serialize with serde. How long their JSON stays readable comes in three tiers:

Versioned: History, with the messages and content it holds. It records its format version, a build reads every version up to its own, and a version bump ships only in a minor release (see Format evolution). Store conversations this way.
Tolerant: TurnEnvelope, the persisted names, and warning codes. The persisted names are the serialized names of capabilities, capability answers, usage counters, embedding limits, error kinds and extraction failures. Golden fixtures pin them and the warning codes, so they change only on purpose, in a minor release. An envelope written by a newer version loads in an older one, with what it does not know read as a fallback (see The turn envelope).
Same-version: everything else, such as ChatRequest, SystemPrompt, ChatResponse, StreamEvent, ChatExchange, StreamTranscript and Error. Their JSON is for logs, recordings and fixtures that the same allama version reads back; a value from another version may not load, or may load with fields left out. For Error the names are pinned (its kind, and the persisted names above), while its shape is same-version.

Content the portable model does not represent arrives as Other content. Content allama later models (audio output, for example) moves from Other to its own variant in a minor release, so match Other only as a fallback, not as the contract.

Compare stored JSON as values, not bytes. Key order in a JSON object, such as ExtraMap (application data, provider data), follows serde_json’s preserve_order feature, which any crate in the build can turn on, so the bytes of the same History differ between builds.

Re-exports§

pub use chat::stream::ChatStream;
pub use chat::stream::ChatStreamExt;
pub use chat::stream::CollectedEvent;
pub use chat::stream::SingleResponseStream;
pub use chat::stream::StreamBlockType;
pub use chat::stream::StreamEvent;
pub use provider_data::ProviderData;
pub use provider_data::ProviderDataEntry;
pub use warning::ReplayState;
pub use warning::RequestField;
pub use warning::SeenWarnings;
pub use warning::Subject;
pub use warning::UnsupportedPolicy;
pub use warning::Warning;
pub use warning::WarningCause;
pub use warning::WarningKind;
pub use schemars;schema

Modules§

adapter: The provider-author toolkit: the steps every adapter shares around encoding a request and decoding a response.
chat: Chat primitives: requests, responses, messages, content, and streaming.
decision: Decision models: typed questions answered with probabilities.
embedding: Embedding primitives.
extractextract: Structured extraction into typed Rust values.
history: Versioned conversation history for persistence.
mockmock: Test doubles for chat, embedding, and decision providers (feature mock).
prelude: Common imports for application code: use allama::prelude::*;.
provider_data: Provider-scoped data attached to content, messages, tools, and responses.
warning: Non-fatal warnings and the unsupported-feature policy.
wrappers: Composable provider wrappers.

Structs§

AssistantMessage: Model output from an earlier turn, replayed as history.
Base64Data: The base64 text of an inline MediaSource: cheap to clone.
CacheBreakpoint: Marks the end of a cacheable prompt prefix.
ChatRequest: A provider-agnostic chat request.
ChatResponse: A provider-agnostic chat response.
ChunkedEmbeddingProgress: What a failed chunked embedding call (embed_chunks or embed_chunked) had done before it failed: where to resume, and the usage billed. It never holds vectors.
Citation: A link from generated text to a source that supports it.
DecisionLimits: Request limits of a decision model, as reported by DecisionProvider::decision_limits.
DecisionRequest: Named questions about one input, answered by a decision model.
DecisionResponse: Answers to a DecisionRequest, by question name.
DynChatProvider: A type-erased chat provider for dynamic dispatch.
DynDecisionProvider: A type-erased decision provider.
DynEmbeddingProvider: A type-erased embedding provider.
EmbeddingChunk: One chunk of a request embedded by embed_chunks: the response for the inputs from offset on, one vector per input.
EmbeddingInput: One item to embed: an ordered list of parts the model combines into one vector.
EmbeddingLimits: Request limits of an embedding model, as reported by EmbeddingProvider::embedding_limits.
EmbeddingRequest: A batch embedding request.
EmbeddingResponse: A batch embedding response.
ErrorContext: Diagnostic context shared by provider-originated errors.
ErrorLog: Redacted logging and telemetry view of an Error.
FallbackValidation: The other side’s dry run in a pair of providers that fails over (see Validation::fallback).
File: File input: a document (PDF, plain text, …), audio, or video.
Image: Image content (input or output).
OtherContent: Content the portable model does not represent, preserved verbatim.
ParseEnumError: A name that is not one of a configuration enum’s values, from its FromStr impl ("high".parse::<ReasoningEffort>()).
PromptCache: Request-level prompt-cache settings.
ProviderToolCall: A tool call the provider executes itself: web search, code execution, a remote MCP server, and so on.
ProviderToolResult: The outcome of a ProviderToolCall.
Reasoning: Reasoning (“thinking”) emitted by the model.
ReasoningConfig: Reasoning configuration.
Refusal: Text in which the model declines to answer, written by the model in place of the answer.
RequestOptions: Provider options attached to a request, keyed by type or namespace.
SerializationError: Wrapper for serialization failures surfaced through Error.
SystemPrompt: One system instruction.
Text: Text content.
ToolArgumentsError: Tool-call arguments that could not be parsed.
ToolCall: A tool call emitted by the model.
ToolDefinition: The definition of a tool the model may call: its name, description, and argument schema. It declares the tool; running it is up to the caller.
ToolMessage: The result of a tool call.
Usage: Token usage reported by a provider.
UserMessage: User input.
Validation: What validating a request found, without sending it.

Enums§

CacheRetention: Preferred lifetime of a cache entry.
CapabilitySupport: Support state for a provider/model capability query.
ChatCapability: Portable chat features a provider or model may support.
CitationSource: The source a Citation points to.
ContentBlock: A block of model output.
ContentPart: A part of user or tool-result input.
DecisionCapability: Portable decision features a provider or model may support.
EmbeddingCapability: Portable embedding features a provider or model may support.
EmbeddingInputType: What embedding inputs are for in asymmetric retrieval.
EmbeddingPart: A part of an EmbeddingInput.
Error: Unified error type for all model operations.
FileKind: The kind of media a File holds, from its media type: what decides the capability a provider needs to take it.
FinishReason: Normalized reason the model stopped generating.
ImageDetail: Requested image fidelity for image input.
MediaSource: Source of media content.
Message: A message in a conversation.
MessageOrigin: Where an error’s constructor message came from.
ReasoningEffort: How much effort the model should spend reasoning.
ReasoningOutput: How much of the model’s reasoning to return.
ResponseFormat: Requested output format.
StreamErrorKind: Kinds of streaming failure.
ToolChoice: How the model should choose tools.
ToolResultContent: Tool output: text, multimodal parts, or a structured JSON object.
UnsupportedStage: Where an Error::Unsupported refusal happened (see Error::unsupported_stage).
UserContent: User message content: plain text or multimodal parts.
ValidationCoverage: How far a Validation went.

Traits§

Capability: A portable capability enum: ChatCapability, EmbeddingCapability or DecisionCapability.
ChatCapabilityResolver: Optional override for a provider’s capability answers.
ChatProvider: Core trait for providers that execute chat requests, whether through a hosted API, a gateway, or an in-process inference runtime.
ChatProviderExt: Convenience methods for every ChatProvider.
DecisionProvider: Providers of decision models: models that answer typed questions about an input with probabilities instead of generating text.
DecisionProviderExt: Convenience methods for every DecisionProvider.
EmbeddingProvider: Providers that turn inputs into vectors.
EmbeddingProviderExt: Convenience methods for every EmbeddingProvider.
ProviderIdentity: Shared provider identity surface for chat, embedding, and decision providers: the provider’s name, and typed lookup of the objects it exposes.
ProviderIdentityExt: Typed access to ProviderIdentity::downcast_any.
ProviderOptions: Implemented by provider-declared option types.

Functions§

expose: Answer a downcast_any query with value when id is its type.

Type Aliases§

ChatValidation: What validating a chat request found.
DecisionValidation: What validating a decision request found.
EmbeddingValidation: What validating an embedding request found.
ExtraMap: A JSON object: the untyped key/value shape used for verbatim provider data.
Result: Convenience alias for std::result::Result<T, E> with Error as the default error type.

Crate allama

Crate allama Copy item path

§Quick start

§Capabilities

§Portability model

§Modules

§Paths

§Cargo features

§Common tasks

§A wrapper stack for providers built directly from adapter crates

§Cross-provider fallback

§Extraction with cost accounting

§Embeddings at scale

§Persisting conversations

§Serialization and stability

Re-exports§

Modules§

Structs§

Enums§

Traits§

Functions§

Type Aliases§

Crate allama