Crate anyllm 

Provider-agnostic chat and embedding primitives and adapters.

The crate root exposes the common day-to-day API surface directly: ChatProvider, ChatRequest, ChatResponse, Tool, EmbeddingProvider, EmbeddingRequest, EmbeddingResponse, and related streaming and wrapper types.

Use prelude when you want a compact import for typical application code. Reach for explicit root imports when writing libraries or examples that need to make advanced types like ChatRequestRecord, ChatResponseRecord, CollectedResponse, or StreamCompleteness obvious at the callsite.

Capabilities

anyllm models each LLM capability as a sibling trait on top of a shared ProviderIdentity super-trait:

• ChatProvider: one-shot and streaming chat completion.
• EmbeddingProvider: batch-oriented text embedding.

Providers implement whichever capabilities they support. A provider that only exposes chat only implements ChatProvider; one that exposes both implements both. Capability support queries share the CapabilitySupport enum across both traits.

Portability Model

anyllm aims to keep the common request/response path provider-agnostic, while still leaving explicit escape hatches for provider-specific features.

The portable chat core is centered on ChatRequest, ChatResponse, Message, ContentBlock, Tool, and the streaming event model. The portable embedding core is centered on EmbeddingRequest, EmbeddingResponse, and EmbeddingCapability, intentionally narrower because the provider overlap on embedding features is narrower.

These types intentionally expose their portable fields directly and pair them with fluent constructors/helpers so application and test code can build and adjust requests and responses without going through opaque builders.

Provider-specific data lives in a few deliberate places: RequestOptions and ResponseMetadata for typed extensions, ExtraMap and extensions fields for portable JSON-shaped extras, and Other enum variants for provider payloads that do not fit the normalized model yet.

This means portability is best-effort rather than absolute. Converting to ChatRequestRecord or ChatResponseRecord preserves the portable core, but typed provider-specific data may be dropped or flattened to JSON.

Wrappers

Four wrappers implement ChatProvider on top of another ChatProvider and compose arbitrarily.

Want	Reach for
Retry the same provider on transient errors	RetryingChatProvider
Swap to a different provider on failure	FallbackChatProvider
OpenTelemetry GenAI spans around each call	TracingChatProvider (tracing feature)
Parse responses into typed Rust structs	ExtractExt::extract (extract feature)

The recommended stacking order, outermost first:

TracingChatProvider::new(
    FallbackChatProvider::new(
        RetryingChatProvider::new(primary),
        RetryingChatProvider::new(fallback),
    ),
)

Rationale:

• Tracing outermost. Spans cover every downstream decision, so a single trace shows the retry attempts and the fallback event.
• Fallback around retry. Each backend gets its own retry budget; exhausted retries on the primary trigger the fallback, and the fallback has its own retry window.
• Retry innermost. Transient errors are handled at the provider that saw them, before bubbling out to the fallback decision.

Structured extraction is orthogonal to the stack. ExtractExt is implemented for every wrapper and for DynChatProvider, so stacked.extract(&request) works on any composition without an explicit ExtractingProvider. Reach for ExtractingProvider only when you want chat() itself to return the extraction-aware response shape rather than calling extract at the call site.

Modules

prelude

Prelude module. Import use anyllm::prelude::* for common application-facing types.

Structs

AssistantMessageRef

Borrowed view returned by Message::as_assistant.

ChatRequest

A provider-agnostic chat completion request.

ChatRequestRecord

Portable, serde-friendly snapshot of a ChatRequest.

ChatResponse

A provider-agnostic chat completion response.

ChatResponseBuilder

Lightweight builder for common ChatResponse test fixtures.

ChatResponseRecord

Portable, serde-friendly snapshot of a ChatResponse.

CollectedResponse

The result of collecting a ChatStream into a ChatResponse via StreamCollector, paired with structural completeness information and any terminal stream error that stopped collection early.

DynChatProvider

A type-erased chat provider for dynamic dispatch.

DynEmbeddingProvider

A type-erased embedding provider for dynamic dispatch.

EmbeddingRequest

A provider-agnostic embedding request.

EmbeddingResponse

A provider-agnostic embedding response.

ErrorLog

Redacted logging/telemetry view of an Error.

Extracted

The result of a structured extraction: a typed value, the underlying provider response, and metadata about how the extraction was performed.

ExtractingProvider

A ChatProvider wrapper that adds orchestrated structured extraction via ExtractExt using a bundled Extractor.

ExtractionMetadata

Metadata about how a structured extraction was performed.

ExtractionRequest

Input for a dedicated structured extraction pass.

Extractor

Orchestrates structured extraction from LLM providers.

FallbackChatProvider

A ChatProvider wrapper that delegates to a fallback provider when the primary provider returns an error that qualifies for built-in failover.

ImageBlockRef

Borrowed view returned by ContentBlock::as_image.

ImagePartRef

Borrowed view returned by ContentPart::as_image.

MockEmbeddingProvider

Deterministic embedding provider for tests.

MockProvider

Deterministic chat provider for tests and downstream fixtures.

MockProviderBuilder

Builder used by MockProvider::build to assemble canned responses and capability metadata.

MockStreamingProvider

Deterministic streaming provider for stream reconstruction and error tests.

MockStreamingProviderBuilder

Builder used by MockStreamingProvider::build to assemble mocked stream transcripts and capability metadata.

MockToolRoundTrip

Two canned responses representing a single tool-call round trip.

OwnedToolCall

Owned representation of a tool call, independent of any ContentBlock.

ReasoningConfig

Configuration for model reasoning/thinking behavior.

RequestOptions

Typed, per-request extension bag keyed by Rust type.

ResponseMetadata

Provider-populated metadata attached to a ChatResponse.

RetryPolicy

Policy controlling retry behavior for transient errors.

RetryingChatProvider

A ChatProvider wrapper that retries non-streaming chat() calls on retryable errors with exponential backoff.

SerializationError

Wrapper for serialization/deserialization failures surfaced through Error.

SingleResponseStream

A zero-allocation stream that emits the normalized event sequence for a fully materialized ChatResponse.

StreamCollector

Accumulates StreamEvents into a CollectedResponse.

StreamCompleteness

Tracks whether a stream was fully received without missing structural events.

SystemOptions

Type-erased bag of per-prompt options keyed by Rust type.

SystemPrompt

A single request-level system instruction.

Tool

A tool (function) definition that the model may call.

ToolCallRef

Borrowed view returned by ContentBlock::as_tool_call.

ToolMessageRef

Borrowed view returned by Message::as_tool.

TracingChatProvider

A wrapper around any ChatProvider that emits tracing spans with OTel Gen AI semantic convention field names for each chat() and chat_stream() call.

TracingContentConfig

Opt-in capture settings for tracing GenAI message content.

Usage

Token usage statistics for a chat completion.

UserMessageRef

Borrowed view returned by Message::as_user.

Enums

CapabilitySupport

Support state for a provider/model capability query.

ChatCapability

Portable chat features that a provider/model may support.

ContentBlock

A single block of content returned by an LLM provider.

ContentPart

A single part within multimodal user content.

EmbeddingCapability

Portable embedding features that a provider/model may support.

Error

Unified error type for all LLM provider operations.

ExtractError

Error produced while extracting structured output

ExtractionMode

Controls how structured data is extracted from a provider response.

FinishReason

Why the model stopped generating.

ImageSource

Source for an image in a content part.

Message

A message in a conversation.

MockResponse

A canned response returned by MockProvider.

MockStreamEvent

One item in a mocked streaming transcript.

ReasoningEffort

How much effort the model should spend on reasoning.

ResponseFormat

Requested response format from the model.

StreamBlockType

The type of content block being streamed.

StreamEvent

A single event in a streaming chat completion response.

ToolChoice

Controls how the model selects tools for a request.

ToolResultContent

Content of a tool result message: either plain text or multimodal parts.

UsageMetadataMode

Interpretation mode for streaming usage metadata

UserContent

Content of a user message: either plain text or multimodal parts.

Traits

ChatCapabilityResolver

Optional resolver used to customize a provider’s chat capability answers.

ChatProvider

Core trait for LLM chat completion providers.

ChatProviderExt

Convenience extension methods for ChatProvider implementors.

ChatStreamExt

Extension trait for consuming a ChatStream into a ChatResponse.

EmbeddingProvider

Core trait for providers that expose a text embedding API.

EmbeddingProviderExt

Convenience extension methods for EmbeddingProvider implementors.

ExtractExt

Extension trait for providers that support structured data extraction.

ProviderIdentity

Shared provider identity surface for chat and embedding providers.

ResponseMetadataType

Marker trait for typed entries stored in ResponseMetadata.

Functions

otel_genai_provider_name

Maps internal provider names onto OTEL GenAI provider semantic-convention values where the spec defines a standard name.

Type Aliases

ChatStream

Boxed stream of normalized chat response events

ExtraMap

Portable JSON object used wherever anyllm needs an untyped, wire-shaped key/value bag.

Result

Convenience alias for std::result::Result<T, Error>.