# genai
**A Native-Protocol Multi-AI Provider Library for Rust**
```toml
# Current release
genai = "0.7"
```
See [v0.7.x](#v070-released)
<div align="center">
<a href="https://crates.io/crates/genai"><img src="https://img.shields.io/crates/v/genai.svg" /></a>
<a href="https://github.com/jeremychone/rust-genai"><img alt="Static Badge" src="https://img.shields.io/badge/GitHub-Repo?color=%23336699"></a>
</div>
[v0.7.x](#v070-released) · [CHANGELOG](CHANGELOG.md) · [Key Features](#key-features) · [Providers](#model-to-adapter-resolution) · [Examples](#examples) · [Chat Options](#chatoptions) · [Usage](#usage) · [TLS Backends](#tls-backends) · [Contribution](#contribution)
`genai` provides a single, ergonomic Rust API for **native-protocol** multi-AI provider access, including Anthropic, OpenAI, Gemini, xAI, Ollama, Groq, and more.
Supports many models and providers out of the box, including **Ollama** for local execution.
Out-of-the-box providers and adapters: `openai`, `openai_resp`, `anthropic`, `gemini`, `gemini_ix`, `omlx`, `ollama`, `ollama_cloud`, `vertex`, `bedrock_api`, `bedrock_sigv4`, `github_copilot`, `opencode_go`, `groq`, `together`, `fireworks`, `cohere`, `nebius`, `mimo`, `deepseek`, `mistral`, `minimax`, `zai`, `zai_coding`, `bigmodel`, `aliyun`, `qwen_cloud`, `baidu`, `moonshot` (moonshot.cn), `kimi` (moonshot.ai), `aihubmix`, `open_router`, `requesty`, `api_route`, `heabsy`, `runcrate`, `cloudgpu`, `atlascloud`, `xai`
Common models from OpenAI, Anthropic, and Gemini can be accessed by name, such as `gpt-6-luna`, `claude-...`, or `gemini-...`, and the appropriate adapter will be selected.
Providers and adapters can also be forced with a namespaced model name via `[name_space]::[model_name]`, for example, `qwen_cloud::qwen3.8-max`
Custom OpenAI-compatible endpoints can be accessed with the built-in `genai_n` adapter. Set `GENAI_{n}_ENDPOINT` and optionally `GENAI_{n}_API_KEY`, then use the `genai_{n}::model_name` namespace:
```text
GENAI_1_ENDPOINT=https://my-host.example.com/v1/
GENAI_1_API_KEY=sk-your-api-key
```
```rust
let client = Client::new()?;
let chat_res = client.exec_chat("genai_1::my-model", chat_req, None).await?;
```
For an OMLX deployment, set `OMLX_ENDPOINT` to the OpenAI-compatible base URL before creating the client. OMLX uses `OMLX_API_KEY` when authentication is required:
```text
OMLX_ENDPOINT=http://my-omlx-host:8000/v1/
OMLX_API_KEY=your-api-key
```
Then target the model with the `omlx::` namespace:
```rust
let client = Client::new()?;
let chat_res = client.exec_chat("omlx::my-model", chat_req, None).await?;
```
Also supports custom endpoints and auth with `ServiceTargetResolver` (see [examples/c06-target-resolver.rs](examples/c06-target-resolver.rs)) to support any other providers.
```rust
// Can talk to any models / providers
let client = Client::new()?;
let question = "Why is the sky red?";
let chat_req = ChatRequest::new(vec![
ChatMessage::system("Answer in one sentence"),
ChatMessage::user(question),
]);
// Model names can even have a reasoning effort suffix, such as "-high", which will be set, and then removed from name when sent to the provider.
let chat_res = client.exec_chat("gpt-5.4-mini-high", chat_req, None).await?;
println!("{}", chat_res.first_text().unwrap_or("NO ANSWER"));
```
[Docs for LLMs](docs/for-llm/api-reference-for-llm.md) | [CHANGELOG](CHANGELOG.md) | [BIG THANKS](BIG-THANKS.md)
## Key Features
- Multi-AI provider/model access optimized per provider: native protocols when available, OpenAI-compatible APIs when appropriate or required, and one common Rust API for OpenAI, OpenAI Responses, Anthropic, Gemini, Ollama, Ollama Cloud, OpenCode Go, Groq, xAI, DeepSeek, Cohere, Together, Fireworks, Nebius, Mimo, Zai, BigModel, Aliyun, Atlas Cloud, Google Vertex, GitHub Copilot, Mistral, Requesty, API Route, Qwen Cloud, Kimi, and OMLX (direct chat and streaming), plus custom `genai_{n}` OpenAI-compatible endpoints (see [examples/c00-readme.rs](examples/c00-readme.rs))
- Image analysis (for OpenAI, Gemini Flash-2, Anthropic) (see [examples/c07-image.rs](examples/c07-image.rs))
- Custom auth/API key (see [examples/c02-auth.rs](examples/c02-auth.rs))
- Model aliases (see [examples/c03-mapper.rs](examples/c03-mapper.rs))
- Custom endpoint, auth, and model identifier (see [examples/c06-target-resolver.rs](examples/c06-target-resolver.rs))
- And much more
[Examples](#examples) | [Thanks](BIG-THANKS.md) | [Library Focus](#library-focus) | [Changelog](CHANGELOG.md) | Provider Mapping: [ChatOptions](#chatoptions) | [Usage](#usage)
## Model to Adapter Resolution
By default, the library resolves the `AdapterKind` (AI provider) based on the model name prefix:
- **OpenAI**: `gpt-*` (most), `o1-*`, `o3-*`, `o4-*`, `chatgpt-*`, `codex-*`
- **OpenAI Responses**: `gpt-5-*`, `gpt-6-*`, `gpt-*` (containing `codex` or `pro`)
- **Anthropic**: `claude-*`
- **Gemini**: `gemini-*`
- **Gemini Interactions API (Argon)**: `gemini-4-argon*` models are routed to the `gemini_ix` adapter. The `gemini_ix::` namespace also forces this adapter.
- **xAI**: `grok-*`
- **DeepSeek**: `deepseek-*`
- **Moonshot**: `moonshot-*`
- **Kimi**: `kimi*` (moonshot.ai)
- **Zai**: `glm-*`
- **Cohere**: `command-*`, `embed-*`
- **Mimo**: `mimo-*`
- **OpenCode Go**: Namespace `opencode_go::` only
- **Mistral AI**: Namespace `mistral::` only
- **Atlas Cloud**: Namespace `atlascloud::` only
- **Qwen Cloud**: Namespace `qwen_cloud::` only, for the managed Qwen Cloud service at qwencloud.com. It uses the international Aliyun-compatible endpoint and `QWEN_CLOUD_API_KEY`.
- **Fireworks**: Models containing `fireworks`
- **Ollama**: Fallback for any other names, defaulting to local Ollama.
### Namespacing (Forcing an Adapter)
You can force a specific adapter by using the `adapter_kind::model_name` syntax. This is the recommended way for many providers and for disambiguating OpenAI-compatible services.
- `groq::openai/gpt-oss-20b` (Forces **Groq** adapter)
- `mistral::mistral-large-4` (Forces **Mistral AI** adapter)
- `together::meta-llama/Llama-3-8b-chat-hf` (Forces **Together** adapter)
- `fireworks::glm-5p1` (for fireworks.ai)
- `ollama_cloud::gemma3:4b` (Forces **Ollama Cloud** adapter)
- `github_copilot::openai/gpt-5.4-mini` (Forces **GitHub Copilot** adapter)
- `nebius::Qwen/Qwen3-235B-A22B` (Forces **Nebius** adapter)
- `aliyun::qwen-plus` (Forces **Aliyun** adapter)
- `qwen_cloud::qwen-plus` (Forces **Qwen Cloud** adapter, managed by qwencloud.com and backed by the international Aliyun-compatible service)
- `vertex::gemini-3.8-flash` (Forces **Google Vertex** adapter)
- `gemini_ix::gemini-3.5-flash` (Forces **Gemini Interactions API** adapter)
- `moonshot::moonshot-v1-8k` (Forces **Moonshot** adapter)
- `baidu::ernie-4.0` (Forces **Baidu** adapter)
- `zai_coding::glm-4.6` (Special namespace for **Zai** coding subscription)
- `opencode_go::minimax-m2.5` (Forces **OpenCode Go** adapter)
- `bedrock_api::anthropic.claude-v2` (Forces **AWS Bedrock** adapter)
- `open_router::google/gemini-2.0-flash-001` (Forces **OpenRouter** adapter)
- `requesty::openai/gpt-4o-mini` (Forces **Requesty** adapter)
- `api_route::gpt-6.1-sol` or `api_route::claude-fable-5-1` (Forces **API Route** adapter)
- `heabsy::qwen38` (Forces **Heabsy** adapter)
- `runcrate::deepseek-ai/DeepSeek-V4.1-Flash` (Forces **Runcrate** adapter)
- `cloudgpu::deepseek-v4.1-flash` (Forces **CloudGPU** adapter)
- `atlascloud::qwen/qwen3.5-flash` (Forces **Atlas Cloud** adapter)
- `genai_1::my-model-7b` (Forces **Custom** adapter with index 1)
For a complete list of `AdapterKind`, see the [AdapterKind enum](src/adapter/adapter_kind.rs).
## v0.7.0 Released
_v0.7.0 release date: 2026-10-19_
See all in [CHANGELOG](CHANGELOG.md) and [v0.7.0 migration guide](docs/migration/migration-v_0_6_to_0_7.md) for more details.
**Big release**: Many fixes, new capabilities, new providers, and minimal API changes.
**Key Breaking Changes:**
- Fallible client initialization: `Client::new()?` and `ClientBuilder::build()?` return `Result<Client>` (no more `Client::default()`).
- `ReasoningEffort::Zero` replaces `ReasoningEffort::None` (`"zero"` keyword, `"none"` backward-compatible alias).
- `Error::HttpError` now carries the failed response `headers`.
- `Tool` adds `custom_format`, and `ChatOptions` adds `raw_frame_sink`, `stream_header_timeout`, and `stream_read_timeout` (use `..Default::default()` or add the fields to struct literals).
- Optional `None` fields are omitted from serialized JSON instead of `null`.
- `JsonSpec::schema_with_additional_properties_false` is removed; adapters now sanitize schemas.
**New Providers & Adapters:**
- Gemini Interactions API (`gemini_ix::`, and `gemini-4-argon*` models mapped to it): stateful conversation sessions (`previous_response_id`, `store`).
- Mistral AI (`mistral::`): Chat Completions with typed thinking content, assistant reasoning replay, and `random_seed` mapping.
- Atlas Cloud (`atlascloud::`), Qwen Cloud (`qwen_cloud::`), Kimi (`kimi::`, `kimi-*`), OMLX (`omlx::`, `omlx-*`).
- Custom adapter (`genai_{n}::`): generic OpenAI-compatible routing via `GENAI_{n}_ENDPOINT` and `GENAI_{n}_API_KEY`.
**Key Capabilities & Improvements:**
- Declarative provider config via `ClientBuilder::append_provider_config` and `ClientConfig::append_provider_config`.
- Stream frame inspection via `ChatFrameSink` and `ChatOptions::with_raw_frame_sink`.
- Thought signature and signed thinking block preservation across tool turns (Anthropic, Gemini, OpenRouter).
- Bedrock Converse: preserve streaming usage from metadata after the stop event, replay signed and redacted reasoning across tool turns, and refine publisher detection, Nova/OpenAI reasoning mappings, and document naming.
- Anthropic prompt caching on tools (`Tool::with_cache_control`) and SSE heartbeat events (`ChatStreamEvent::Heartbeat`).
- OpenTelemetry GenAI semantic conventions instrumentation behind the `otel` feature flag.
## Examples
- [examples/c00-readme.rs](examples/c00-readme.rs) - Quick overview code with multiple providers and streaming.
- [examples/c01-conv.rs](examples/c01-conv.rs) - Shows how to build a conversation flow.
- [examples/c02-auth.rs](examples/c02-auth.rs) - Demonstrates how to provide a custom `AuthResolver` to supply auth data, such as `api_key`, per adapter kind.
- [examples/c03-mapper.rs](examples/c03-mapper.rs) - Demonstrates how to use a custom `ModelMapper` to customize model mapping.
- [examples/c04-chat-options.rs](examples/c04-chat-options.rs) - Demonstrates how to set chat generation options such as `temperature` and `max_tokens` at the client level, for all requests, and at the per-request level.
- [examples/c05-model-names.rs](examples/c05-model-names.rs) - Shows how to get model names per `AdapterKind`.
- [examples/c06-target-resolver.rs](examples/c06-target-resolver.rs) - For custom auth, endpoint, and model.
- [examples/c07-image.rs](examples/c07-image.rs) - Image analysis support
<br />
<a href="https://www.youtube.com/playlist?list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj"><img alt="Static Badge" src="https://img.shields.io/badge/YouTube_JC_AI_Playlist-Video?style=flat&logo=youtube&color=%23ff0000"></a>
<br />
- [genai ModelMapper code demo (v0.1.7)](https://www.youtube.com/watch?v=5Enfcwrl7pE&list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj)
- [genai introduction (v0.1.0)](https://www.youtube.com/watch?v=uqGso3JD3eE&list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj)
- **genai live coding, code design, and best practices**
- [Adding **Gemini** Structured Output (vid-0060)](https://www.youtube.com/watch?v=GdFsqLJ1_pE&list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj)
- [Adding **OpenAI** Structured Output (vid-0059)](https://www.youtube.com/watch?v=FpoNbQMhAH8&list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj)
- [Splitting the JSON value extension trait into its own public crate, value-ext](https://www.youtube.com/watch?v=OS5KOz9y7Cg&list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj) [value-ext](https://crates.io/crates/value-ext)
- [(part 1/3) Module, Error, constructors/builders](https://www.youtube.com/watch?v=XCrZleaIUO4&list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj)
- [(part 2/3) Extension Traits, Project Files, Versioning](https://www.youtube.com/watch?v=LRfDAZfo00o&list=PL7r-PXl6ZPcBcLsBdBABOFUuLziNyigqj)
- [(part 3/3) When to Async? Project Files, Versioning strategy](https://www.youtube.com/watch?v=93SS3VGsKx4&list=PL7r-PXl6ZPcCIOFaL7nVHXZvBmHNhrh_Q)
## Library Focus:
- Focuses on standardizing chat completion APIs across major AI providers while preserving provider-specific strengths.
- Native implementation without per-service SDK dependencies.
- Reason: genai uses each provider's native protocol when available, so features such as reasoning controls, thinking budgets, streaming metadata, and multimodal inputs can be represented more completely. When a provider primarily exposes an OpenAI-compatible API, genai uses that compatibility layer instead. Managing these protocol differences at the adapter layer is simpler and more scalable than dealing with multiple SDKs.
- Prioritizes ergonomics and commonality, while depth is secondary. (If you require a complete client API, consider using [async-openai](https://crates.io/search?q=async-openai) and [ollama-rs](https://crates.io/crates/ollama-rs); both are excellent and easy to use.)
- This library focuses on text chat, vision, and function calling APIs. (If you require a complete client API, consider using [async-openai](https://crates.io/search?q=async-openai) and [ollama-rs](https://crates.io/crates/ollama-rs); both are excellent and easy to use.)
## ChatOptions
- **(1)** - **OpenAI-compatible** notes
- Models: OpenAI, DeepSeek, Groq, xAI, Mimo, Together, Fireworks, Nebius, Zai, AIHubMix, Atlas Cloud
| Property | OpenAI Compatibles (*1) | Anthropic | Gemini `generationConfig.` | Cohere |
| ------------- | ----------------------- | --------------------------- | -------------------------- | ------------- |
| `temperature` | `temperature` | `temperature` | `temperature` | `temperature` |
| `max_tokens` | `max_tokens` | `max_tokens` (default 1024) | `maxOutputTokens` | `max_tokens` |
| `top_p` | `top_p` | `top_p` | `topP` | `p` |
## Usage
| Property | OpenAI Compatibles (1) | Anthropic `usage.` | Gemini `usageMetadata.` | Cohere `meta.tokens.` |
| --------------------------- | --------------------------- | ----------------------- | -------------------------- | --------------------- |
| `prompt_tokens` | `prompt_tokens` | `input_tokens` (added) | `promptTokenCount` (2) | `input_tokens` |
| `completion_tokens` | `completion_tokens` | `output_tokens` (added) | `candidatesTokenCount` (2) | `output_tokens` |
| `total_tokens` | `total_tokens` | (computed) | `totalTokenCount` (2) | (computed) |
| `prompt_tokens_details` | `prompt_tokens_details` | `cached/cache_creation` | N/A for now | N/A for now |
| `completion_tokens_details` | `completion_tokens_details` | N/A for now | N/A for now | N/A for now |
- **(1)** - **OpenAI-compatible** notes
- Models: OpenAI, DeepSeek, Groq, xAI, Mimo, AIHubMix, Atlas Cloud
- For **Groq**, the property is `x_groq.usage.`
- Usage availability can vary by provider and streaming mode.
- `prompt_tokens_details` and `completion_tokens_details` will have the value sent by the compatible provider, or `None`
- **(2)**: **Gemini** tokens
- Right now, with the [Gemini Stream API](https://ai.google.dev/api/rest/v1beta/models/streamGenerateContent), it’s not clear whether usage for each event is cumulative or must be summed. It appears to be cumulative, meaning the last message shows the total number of input, output, and total tokens, so that is the current assumption. See [possible tweet answer](https://twitter.com/jeremychone/status/1813734565967802859) for more info.
## Usage examples
- [AIPack](https://aipack.ai) - Check out [AIPack](https://aipack.ai), which wraps this **genai** library into an agentic runtime to run, build, and share AI Agent Packs. See [`pro@coder`](https://news.aipack.ai/p/procoder-v052-demo-workbench) for a simple example of how I use AI PACK/genai for production coding.
- [zcoder](https://zcoder.run) - I am also in the process of building [zcoder](https://zcoder.run), which will be a parallel-first coding harness.
> Note: Feel free to send me a short description and a link to your application or library that uses genai. I'm happy to add it.
## TLS Backends
TLS is selectable via cargo features. The default works out of the box and is right for almost everyone.
| You want… | Use |
| -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A backend that just works (default) | nothing — `rustls-tls` (rustls + aws-lc-rs + OS trust store) |
| System trust policy / OpenSSL or SChannel | `genai = { default-features = false, features = ["native-tls", "system-proxy"] }` |
| ring / FIPS / no `aws-lc-sys` (lean musl & cross builds) | `genai = { default-features = false }`, add your own `reqwest = { features = ["rustls-no-provider"] }` + a provider (e.g. `rustls` with `ring`), and call `CryptoProvider::install_default()` at startup |
| Static OpenSSL | `genai = { default-features = false }` + your own `reqwest = { features = ["native-tls-vendored"] }` |
| Custom root CA / mTLS / fully custom client | build your own `reqwest::Client` and inject it via `with_reqwest(...)` |
`rustls-tls` and `native-tls` are mutually exclusive. When selecting `native-tls`, set `default-features = false` (as shown above) — otherwise both backends compile and the build fails with a `compile_error!` telling you to do so.
`rustls-tls` reads the OS trust store via `rustls-platform-verifier`, so enterprise/custom CAs work without `native-tls`, and it needs no system OpenSSL. Note the default `aws-lc-rs` provider builds `aws-lc-sys` (a C/assembly crate); for a leaner static-musl or cross build, drop it with `default-features = false` and use the `ring` provider as in the table above.
For full control, build a `reqwest::Client` yourself and inject it:
```rust
let reqwest_client = reqwest::Client::builder()
// .add_root_certificate(...), client identity for mTLS, custom CryptoProvider, etc.
.build()?;
let client = genai::Client::builder()
.with_reqwest(reqwest_client)
.build()?;
```
If you set `default-features = false` on `genai` without enabling a TLS feature, add `reqwest` with a TLS feature directly to your own `Cargo.toml` (or use the `rustls-no-provider` path and install a `CryptoProvider`) — otherwise HTTPS requests fail at runtime.
The default `system-proxy` feature turns on reqwest's `system-proxy`, which reads the macOS and Windows system proxy settings when a client is built. Cargo unifies features, so it applies to every `reqwest` client in the binary. With `default-features = false` it is off; add `features = ["system-proxy"]` to keep it. Proxy environment variables (`HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`) are honored either way.
## Contribution
Contributions are welcome. To keep review fast and the codebase consistent, please follow these guidelines.
- Keep your PR focused on one problem or one enhancement, or on very closely related changes. Small, focused PRs are easier to review and faster to merge.
- This crate follows a zero-panic approach, so avoid `.unwrap()`, `.expect(...)`, and similar panic-prone calls in the production code path, and avoid array indexing with `[]` as much as possible.
- For now, `.unwrap()` and `.expect(...)` are tolerated in tests and examples, but they should be avoided going forward and will be changed soon. Favor `.ok_or("...")?` and other non-panicking alternatives.
- When changing a provider or adapter implementation behavior, updating or adding a Yakbak cassette is appreciated when it makes sense.
- Using AI is completely fine, as long as you fully review and control the end result before submitting the PR.
- Concise, to-the-point description.
- Clean and focused code.
- Emoticonless, especially in code.
- Do not update `CHANGELOG.md`, `BIG-THANKS.md`, or the migration docs under `docs/migration/` in your PR. The maintainer handles those in a single pass so the formatting, wording, and ordering stay consistent across releases. Describe the change in your PR instead, and call out anything user-visible or breaking so it can be captured accurately.
- If CHANGELOG.md or migration files are erroneous, submitting a PR to rectify them is a good approach.
## Links
- crates.io: [crates.io/crates/genai](https://crates.io/crates/genai)
- GitHub: [github.com/jeremychone/rust-genai](https://github.com/jeremychone/rust-genai)