Crate rust_llm 

rust_llm

One API for chat, tools, agents, structured output, streaming, embeddings, images, batches, MCP, and judgments across LLM providers.

let mut chat = rust_llm::chat_with("claude-opus-5-5")?;

let answer = chat.ask("What's the best way to learn Rust?").await?;
println!("{}", answer.content());

Features

• Chat: conversations with instructions, model switching, temperature, provider options, headers, request hooks, callbacks, fallbacks, and cancellation. Start with chat() or chat_with.
• Tool: tools the model can call, with Parameter declarations or a schemars schema (schema_for), approvals, and progress. ProviderTools run on the provider.
• Agent: a chat configuration declared once and reused.
• Streaming (Chat::ask_stream), structured output (Chat::with_schema_for), and extended thinking (ThinkingConfig).
• Attachments (Attachment) and provider Files APIs (upload, download).
• embed, paint, batch() and embed_later, judge() and Judge.
• MCP servers (mcp, Mcp).
• Per-attempt Tokens and Cost on every result.

Providers: OpenAI, Anthropic, Gemini, DeepSeek, Mistral, OpenRouter, xAI, Perplexity, Ollama, Ollama Cloud, GPUStack, Hetzner, and TypeSafe. Configure them with configure; keys such as OPENAI_API_KEY are read from the environment automatically.

Guides

The user guides cover each feature in depth. RustLLM is a port of RubyLLM 2.1, and keeps its names where Rust allows (ask, ask_later, complete, with_tools, before_tool_call, …), with ?-suffixed predicates spelled is_*; RubyLLM vs RustLLM maps one to the other.

Re-exports

pub use agent::Agent;

pub use attachment::Attachment;

pub use attachment::Resolution;

pub use batch::Batch;

pub use batch::BatchResult;

pub use batch::BatchStatus;

pub use batch::EmbeddingRequest;

pub use batch::batch;

pub use batch::embed_later;

pub use cached_content::CacheOptions;

pub use cached_content::CachedContent;

pub use cached_content::Ttl;

pub use cached_content::cache;

pub use chat::CancelHandle;

pub use chat::Chat;

pub use chat::Fallback;

pub use chat::FallbackAttempt;

pub use config::Config;

pub use config::config;

pub use config::configure;

pub use context::Context;

pub use context::context;

pub use cost::Cost;

pub use embedding::EmbedOptions;

pub use embedding::Embedding;

pub use embedding::SparseVectors;

pub use embedding::Vectors;

pub use embedding::embed;

pub use error::Error;

pub use error::ErrorKind;

pub use error::Result;

pub use files::DownloadedFile;

pub use files::FileOptions;

pub use files::UploadOptions;

pub use files::UploadedFile;

pub use files::download;

pub use files::upload;

pub use image::Image;

pub use image::Images;

pub use image::PaintOptions;

pub use image::paint;

pub use instrumentation::Instrumenter;

pub use instrumentation::instrument;

pub use judge::Answer;

pub use judge::Dynamic;

pub use judge::Judge;

pub use judge::JudgeOptions;

pub use judge::Judgment;

pub use judge::QuestionType;

pub use judge::judge;

pub use judge::list_judgment_models;

pub use mcp::InputRequest;

pub use mcp::Mcp;

pub use mcp::McpResult;

pub use mcp::McpTool;

pub use message::Chunk;

pub use message::Citation;

pub use message::FinishReason;

pub use message::Message;

pub use message::Role;

pub use message::Thinking;

pub use message::ToolCall;

pub use message::UsageEntry;

pub use message::UsageStatus;

pub use model::Model;

pub use models::models;

pub use moderation::ModerateOptions;

pub use moderation::Moderation;

pub use moderation::ModerationInput;

pub use moderation::ModerationResult;

pub use moderation::moderate;

pub use ocr::Ocr;

pub use ocr::OcrOptions;

pub use ocr::Page as OcrPage;

pub use ocr::ocr;

pub use progress::Progress;

pub use prompt::Prompt;

pub use prompt::render_prompt;

pub use protocols::Caching;

pub use protocols::ToolCalls;

pub use protocols::ToolChoice;

pub use provider_tools::ProviderTool;

pub use providers::ProtocolName;

pub use providers::Provider;

pub use request_shape::RequestShape;

pub use rerank::Rerank;

pub use rerank::RerankOptions;

pub use rerank::RerankResult;

pub use rerank::rerank;

pub use search_results::SearchResults;

pub use speech::SpeakOptions;

pub use speech::Speech;

pub use speech::SpeechChunk;

pub use speech::speak;

pub use speech::speak_stream;

pub use thinking::Display as ThinkingDisplay;

pub use thinking::ThinkingConfig;

pub use tokenization::Tokenization;

pub use tokenization::TokenizeOptions;

pub use tokenization::tokenize;

pub use tokens::Tokens;

pub use tool::FnTool;

pub use tool::Parameter;

pub use tool::SharedTool;

pub use tool::Tool;

pub use tool::ToolError;

pub use tool::ToolResult;

pub use tool::schema_for;

pub use transcription::TranscribeOptions;

pub use transcription::Transcription;

pub use transcription::TranscriptionChunk;

pub use transcription::transcribe;

pub use transcription::transcribe_stream;

pub use video::AnimateOptions;

pub use video::Video;

pub use video::VideoJob;

pub use video::VideoSource;

pub use video::VideoStatus;

pub use video::animate;

pub use video::animate_later;

pub use workflow::Workflow;

pub use workflow::workflow;

Modules

accounting

Port of lib/ruby_llm/accounting/usage.rb: the per-attempt usage Tracker, the Entry readers it relies on (the UsageEntry struct itself lives in message.rs), and the Accounting::Usage::Result storage shared by results that carry attempt accounting.

agent

Port of lib/ruby_llm/agent.rb: reusable assistants declared once.

attachment

Port of lib/ruby_llm/attachment.rb and files/mime_type.rb.

batch

Port of lib/ruby_llm/batch.rb, lib/ruby_llm/embedding_request.rb, the batch methods of lib/ruby_llm/provider.rb, and the batch protocols: protocols/anthropic/batches.rb, protocols/openai/batches.rb (with responses/batches.rb, chat_completions/batches.rb, chat_completions/embedding_batches.rb), protocols/gemini/batches.rb, protocols/gemini/embedding_batches.rb, protocols/openrouter/batches.rb, providers/mistral/chat_completions/batches.rb, and providers/xai/chat_completions/batches.rb.

cached_content

Port of lib/ruby_llm/cached_content.rb, RubyLLM.cache, and lib/ruby_llm/protocols/gemini/caches.rb: provider-side prompt caches (Gemini’s cachedContents).

chat

Port of lib/ruby_llm/chat.rb: the conversation loop.

config

Port of lib/ruby_llm/configuration.rb.

context

Port of lib/ruby_llm/context.rb and RubyLLM.context: an isolated configuration scope with the same entry points as the crate root, reading its own Config copy instead of the global.

cost

Port of lib/ruby_llm/cost.rb for text tokens.

embedding

Port of lib/ruby_llm/embedding.rb with the OpenAI-compatible and Gemini embedding protocols.

error

Port of lib/ruby_llm/error.rb and the status mapping in lib/ruby_llm/transport/error_middleware.rb.

evaluation

Evaluations: run your application against a dataset of cases and grade each result. Port of lib/ruby_llm/evaluation.rb and lib/ruby_llm/evaluation/* (case, dataset, evaluator, evidence, report, result, reviewer, runner, trial, assertions).

files

Port of lib/ruby_llm/uploaded_file.rb, downloaded_file.rb, protocols/files.rb with protocols/{openai,anthropic,gemini,mistral,xai,deepseek,openrouter,perplexity}/files.rb, and the auto-upload half of protocol.rb (preprocess_message, preprocess_attachment, provider_upload) with each protocol’s thresholds from its chat.rb.

image

Port of lib/ruby_llm/image.rb (RubyLLM.paint), Protocol#paint/post_image/parse_image_responses in lib/ruby_llm/protocol.rb, and the image seams in protocols/chat_completions/images.rb (OpenAI Images API), protocols/gemini/images.rb, providers/xai/images.rb, and providers/openrouter/images.rb.

instrumentation

Port of lib/ruby_llm/support/instrumentation.rb, RubyLLM.instrument, and config.instrumenter.

judge

Judgments: typed questions about your data, answered by a System One model such as TypeSafe’s Jev, a local decision model through Ollama, or OpenAI Decisions. Port of lib/ruby_llm/judge.rb, judge/question.rb, judgment.rb, probability.rb, choice.rb, score.rb, protocols/system_one/*.rb, and providers/ollama/judgments.rb; OpenAI Decisions is in decisions.

mcp

Port of lib/ruby_llm/mcp.rb: a client for a Model Context Protocol server.

message

Port of message.rb, chunk.rb, tool_call.rb, thinking.rb, citation.rb, server_tool_call.rb, and the usage Entry from accounting/usage.rb.

model

Port of lib/ruby_llm/model.rb and lib/ruby_llm/model/pricing*.rb.

models

Port of lib/ruby_llm/models.rb, models/lookup.rb, and models/aliases.rb: the bundled registry, alias resolution, and provider preference. Refreshing lives in refresh, registry files in registry.

moderation

Port of lib/ruby_llm/moderation.rb (RubyLLM.moderate), Protocol#moderate, and lib/ruby_llm/protocols/chat_completions/moderation.rb, which every Chat Completions and Responses provider inherits.

ocr

Port of lib/ruby_llm/ocr.rb (RubyLLM.ocr), Protocol#ocr, and lib/ruby_llm/providers/mistral/ocr.rb, the only OCR seam among the providers RustLLM supports.

open_telemetry

Port of lib/ruby_llm/open_telemetry.rb and lib/ruby_llm/open_telemetry/attributes.rb: optional OpenTelemetry tracing through the instrumentation events.

progress

Port of lib/ruby_llm/progress.rb, support/progress_reporter.rb, and support/cancellation.rb.

prompt

Port of lib/ruby_llm/prompt.rb and lib/ruby_llm/prompt/context.rb: prompt templates under app/prompts, rendered with locals.

protocols

Port of lib/ruby_llm/protocol.rb, protocol/streaming.rb, and protocol/stream_accumulator.rb.

provider_tools

Port of lib/ruby_llm/tools/provider_tools.rb, Protocol#apply_provider_tools, and the SERVER_TOOL_ALIASES tables of the protocols this crate speaks: tools that run on the provider’s servers, such as web search, code execution, or a remote MCP server.

providers

Port of lib/ruby_llm/provider.rb and lib/ruby_llm/providers/*.rb: where each service lives, how it authenticates, and which wire protocol it speaks for a model.

request_shape

Port of lib/ruby_llm/request_shape.rb and lib/ruby_llm/request_shape/{part,turn,tool_round,problem}.rb.

rerank

Port of lib/ruby_llm/rerank.rb (RubyLLM.rerank), Protocol#rerank, and lib/ruby_llm/protocols/chat_completions/rerank.rb, the Jina-style endpoint OpenRouter and GPUStack serve. Cohere’s and Bedrock’s rerank protocols belong to providers RustLLM does not port.

search_results

Port of lib/ruby_llm/search_results.rb: documents a tool returns so the model can cite each source. Protocols that support it (Anthropic) render them in their citation format; cited passages come back on Message::citations.

speech

Port of lib/ruby_llm/speech.rb (RubyLLM.speak), speech_chunk.rb, Protocol#speak / stream_speech / stream_speech_response in lib/ruby_llm/protocol.rb, protocol/binary_streaming.rb, and the speech seams in protocols/chat_completions/speech.rb (OpenAI and every OpenAI-compatible provider), protocols/gemini/speech.rb, providers/mistral/speech.rb, providers/xai/speech.rb, providers/openrouter/speech.rb, and providers/gpustack/speech.rb.

support

Port of lib/ruby_llm/support/process_cache.rb.

thinking

Port of Thinking::Config and Thinking::Controls from lib/ruby_llm/thinking.rb: with_thinking options resolved against what the model’s registry entry accepts.

tokenization

Port of lib/ruby_llm/tokenization.rb (RubyLLM.tokenize) with protocols/xai/tokenization.rb and protocols/gpustack/tokenization.rb, plus the input-token counting seams behind Chat#count_tokens: render_count_tokens_payload/count_tokens_url/parse_count_tokens_response in protocols/anthropic/chat.rb, protocols/gemini/chat.rb, and protocols/responses/token_counting.rb.

tokens

Port of lib/ruby_llm/tokens.rb.

tool

Port of lib/ruby_llm/tool.rb.

tool_search

Tool search with deferred tool loading: lib/ruby_llm/chat/tool_search.rb, the apply_tool_search/tool_search_block? halves of protocols/anthropic/tools.rb and protocols/responses/tools.rb, and each protocol’s portable_raw_content (protocols/anthropic/chat.rb, protocols/responses/chat.rb).

transcription

Port of lib/ruby_llm/transcription.rb (RubyLLM.transcribe), transcription_chunk.rb, Provider#transcribe and Protocol#transcribe / build_audio_file_part / audio_file_name in lib/ruby_llm/protocol.rb, and the transcription seams in protocols/chat_completions/transcription.rb (OpenAI and every OpenAI-compatible provider), protocols/gemini/transcription.rb (generateContent), protocols/gemini/file_transcription.rb with protocols/interactions/transcription.rb (Gemini’s dedicated transcription models), protocols/openrouter/transcription.rb, providers/mistral/transcription.rb, providers/xai/transcription.rb, and providers/gpustack/transcription.rb. WebSocket transcription lives in gemini_live (protocols/gemini/live_transcription.rb) and xai_streaming (protocols/xai/streaming_transcription.rb), over WebsocketConnection, with wav_audio (transcription/wav_audio.rb) reading the PCM format.

transport

Port of lib/ruby_llm/transport/connection.rb, error_middleware.rb, and the SSE handling in protocol/streaming.rb.

video

Port of lib/ruby_llm/video.rb (RubyLLM.animate), lib/ruby_llm/video_job.rb (RubyLLM.animate_later), Protocol#animate_later/refresh_video_job, and the video seams in providers/xai/videos.rb, providers/openrouter/videos.rb, protocols/gemini/videos.rb, and protocols/gpustack/videos.rb.

workflow

Port of lib/ruby_llm/workflow.rb and RubyLLM.workflow: named, instrumented regions of ordinary async code.

Macros

evaluates

One #[test] per case: evaluates!(formatting_evaluation(); correct => "correct", regression => "regression"); where the first expression builds the Evaluation. Each failing test prints the report.

Constants

VERSION

RubyLLM::VERSION of the port’s source.

Functions

chat

RubyLLM.chat with the configured default model.

chat_with

RubyLLM.chat(model:).

count_tokens

RubyLLM.count_tokens(text, model:, provider:): input tokens for one user message.