Intro
llmao is an abstraction layer for interfacing with different LLM providers.
The idea is that llmao creates a unified and generic way of communicating
between LLM providers. It defines the traits and types but does not include
provider implementations, you must bring your own provider client and implement
these traits for it. With that being said, if you just want high-level crates
that already implement provider specific protocols, see
high level crates.
Design Goals
llmao is heavily inspired by the amazing
embedded-hal crate, and so
therefore will emulate the same sort of mental model as well as share similar
design goals.
- Async by default (for now), since these requests are likely going over the wire using HTTPS, async will be the default implementation for these traits. Ideally, in the future, I'd like to add blocking variants for the async functions.
- Where possible must not be tied to a specific asynchronous model. This mean it should work regardless of async runtimes (tokio, async-std, smol).
- Must be minimal, and thus easy to implement and zero cost, yet highly composable.
- Serve as a foundation for creating an ecosystem of provider-agnostic LLM operations. An operation in this case, simply means calling an API endpoint.
- Trait methods must be fallible so that they can be used in any possible situation.
- Must erase provider-specific details. No OpenAI parameters, Claude-specific formatting, or provider endpoints that would leak into this trait API.
- Must use associated types for errors, allowing each provider implementation to define its own error type without forcing allocation or type erasure.
- Schema/structured output support should use the provider's native capabilities (JSON schema parameter).
Out of Scope
- Agentic specific implementations
- Model Context Protocol (MCP) specific implementations
High Level Crates
- 0xPlaygrounds/rig -
rig-coreimplements their own generic trait system that allows you to build and configure their suppported clients.